This Linux command is Really Useful! `pv` aka Pipe Viewer

Summary of the video (tech concepts + features of pv)

The video is a tutorial-style deep dive into the Linux pv command (“Pipe Viewer”), explaining how it can be inserted into shell pipelines to monitor data transfer. pv behaves like a Unix filter: it reads from stdin and writes to stdout, but it can also display statistics, including progress bars, rate, bytes transferred, and ETA.

Core capabilities highlighted

• Progress display for pipeline data (e.g., watching cat-like output)
• Transfer statistics: amount read/written, current throughput (rate)
• ETA and percentage when size is known
• Rate limiting (slowing down a pipeline to a specified speed)
• Silent mode (quiet mode) while still providing throttling behavior
• Timer-only mode to measure elapsed time for long-running piped commands
• Support for multiple concurrent pv instances with cursor positioning / named displays
• Compatibility with typical pipeline tools such as tar, gzip, ssh, dd, and netcat-style flows

---

Examples and what they demonstrate

1) Basic progress bar in a pipeline

• Idea: pipe a large file through pv to show how fast data is being written.
• Example concept: cat a big file → pipe through pv.

2) Progress bar when the producer is slow (and redirecting output)

• Uses a “slowcat” concept to show that pv can display throughput over time.
• If terminal output becomes messy, it’s common to redirect output to:
  • a file
  • /dev/null
  • or other sinks

3) Why ETA matters: pv can’t always know the input size

• pv can’t estimate progress duration if it doesn’t know the total size.
• If the input is from a pipe, pv doesn’t automatically know byte count.
• The video shows providing size explicitly using a -s style argument (narration mentions a -d-size-like approach / passing a size denominator).
• Result: pv can show percentage + ETA (e.g., “16 hours” remaining).

4) -L “line mode” for text (instead of bytes)

• Introduces line-aware progress:
  • default pv is byte-oriented
  • with -L, pv switches to line-based interpretation
• A “slowcat” alias concept is explained as an alias to pv with flags:
  • line mode enabled
  • and a small transfer rate limit, where “5” means lines per second, not bytes per second

5) Quiet throttling: rate limiting without progress output

• Demonstrates quiet mode (-q):
  • pv can act as a rate limiter even when you don’t want progress text.
• Use case: pipe a file into ssh so the transfer is throttled but remains streamed.

6) Streaming to a remote host with a live progress bar

• Example pipeline concept:
  • cat / tar → pv (progress) → ssh receiver → write to a file
• Shows that pv progress remains useful even if you don’t store data locally first.
• Verification: run a command like word count -l on the remote output after completion to confirm the full file arrived.

7) Pipeline compression workflows without intermediate files

• Shows pipelines like:
  • tar to stdout → pv → gzip → (optionally another pv) → write output
• Demonstrates progress tracking across multi-stage pipelines.
• Also mentions an illustrative demo where progress is visible but output looks like “binary/Hacker-mode” when piped into something like hexdump (mostly for effect).

8) When total size is unknown (e.g., tar to gzip)

• Key observation: if pv can’t determine total size (common in streaming flows like tar | gzip):
  • percent/ETA may be missing or not meaningful
  • but you still get live transfer statistics

9) Using pv with dd-like scenarios and comparing status/progress

• Notes that newer dd includes status=progress, but pv can provide progress in other pipeline contexts.
• Example concept:
  • dd if=/dev/zero → pv → dd to /dev/null
• Also notes pv can throttle network-style pipes (e.g., netcat-style transfer).

10) Timer mode: pv -t

• Highlights a lesser-known flag: pv -t.
• Acts as an elapsed timer as long as stdin stays open.
• Example: pipe a long-running command into pv -t to show runtime.

---

Multiple pv instances: cursor mode and naming

Cursor/terminal alignment problem

• Running multiple pv instances in the same terminal can cause overlapping output.
• pv supports:
  • cursor mode (-c)
  • and naming (-n)

Using -c and -n to coordinate displays

• Demonstrated approach:
  • multiple pv -c -n <name> instances
• pv repositions the cursor so each progress bar updates in its own “slot”.

Ordering / race condition note

• The author warns that pipeline processes may start concurrently, so the “slot order” PV claims may not be deterministic.
• Recommendation: use naming (-n) when using cursor mode so bars remain consistently tracked by label.

Hacky sleep-based race workaround (not recommended generally)

• The video shows a demonstration using sleep in subshells to influence which PV attaches first.
• PSA: don’t “fix” race conditions with sleeps in real systems—this was only for illustration.

---

How PV coordinates multiple progress bars (source-code deep dive)

The video includes a nerdy section compiling and inspecting PV’s source code to explain multi-instance behavior:

• In cursor mode, PV coordinates updates across processes.
• If available, PV uses shared memory (IPC / shared memory segment) to coordinate where each instance should draw its status bar.
• It stores:
  • an original Y-coordinate
  • and then applies offsets to place each display
• If terminal locking isn’t supported, PV falls back to a lock file.
• The author also shows enabling debug logging, compiling with debug support, and running PV with debug flags to observe:
  • terminal lock acquisition
  • cursor coordination

Key mechanism described

• Lock terminal
• Attach / coordinate via shared memory
• Write cursor update
• Release lock

This locking approach is presented as the reason PV avoids partial or overlapping cursor writes (cursor movement + text) across multiple PV processes.

---

Product/tooling review or tutorial takeaways

• pv is presented as a general-purpose pipeline tool for:
  • monitoring long-running transfers
  • adding progress bars without modifying other programs
  • throttling streams (including over SSH) via -q
  • line-based monitoring for text via -L
  • simple elapsed timing via -t
• The video strongly encourages:
  • reading PV’s man page (“a ton of stuff” beyond what’s shown)
  • using cursor mode (-c) carefully with multiple instances, preferably with -n.

---

Main speakers / sources (as indicated)

• Speaker: “Dave” (narrator references “Dave” and delivers a first-person tutorial style)
• Primary tool/docs source: pv command documentation and PV source code (downloaded/compiled from its project repository, referenced as being on a website such as Codeberg)