Pipes connect processes, not strings

This looks straightforward:

printf 'one\ntwo\nthree\n' | wc -l

The first command produces:

one
two
three

The second counts three lines.

My first mental model was something like:

run printf
    │
    ▼
collect its output as a string
    │
    ▼
give that string to wc

That is close enough to predict the visible result.

It also hides most of what a shell pipeline actually does.

Bash does not normally wait for the first command to finish and collect one completed string. It creates a communication channel, connects two executing pipeline stages and lets bytes flow between them.

The more useful picture is:

printf stage
      │
      │ standard output
      ▼
     pipe
      │
      │ standard input
      ▼
   wc stage

The vertical bar is process plumbing, not a string-passing operator.

These examples assume Bash on Linux with common GNU command-line tools unless noted otherwise.

What Bash actually connects

Bash calls this:

command1 | command2

a pipeline.

The shell connects the standard output of command1 to the standard input of command2.

So:

printf 'one\ntwo\nthree\n' | wc -l

means something conceptually like:

printf

fd 1
 │
 ▼
write end of pipe


read end of pipe
 │
 ▼
fd 0

wc

The two commands retain their familiar interfaces:

printf
   │
   └── writes bytes to stdout

wc
   │
   └── reads bytes from stdin

Neither command needs to know that Bash placed a pipe between them.

printf writes to file descriptor 1.

wc reads from file descriptor 0.

The shell arranges where those descriptors lead.

A pipeline stage is not necessarily a separate external executable. It may also be a shell builtin, function or compound command such as a loop. Bash still gives each stage its own execution context and connects the relevant file descriptors.

The shell creates the connection first

A simplified pipeline setup looks like this:

Bash
 │
 ├── creates a pipe
 │      ├── read end
 │      └── write end
 │
 ├── starts command1
 │      └── connects stdout to write end
 │
 ├── starts command2
 │      └── connects stdin to read end
 │
 ├── closes pipe ends it does not need
 │
 └── waits for the pipeline

The operating system represents each end of the pipe using a file descriptor:

pipefd[0]    read end
pipefd[1]    write end

For the first stage, Bash arranges:

stdout ──► pipe write end

For the second:

stdin ◄── pipe read end

A pipeline is another answer to:

Where does this process’s output actually go?

The stages can run at the same time

The first stage does not usually need to finish before the second starts.

Try:

for number in 1 2 3 4 5; do
    printf '%s\n' "$number"
    sleep 1
done |
while IFS= read -r number; do
    printf 'received: %s\n' "$number"
done

The result appears gradually:

received: 1
received: 2
received: 3
received: 4
received: 5

The producer writes one line. The consumer reads it. Then the producer waits before creating the next one.

Conceptually:

PRODUCER                  CONSUMER

write "1\n" ─────────────► read "1\n"
sleep                      process 1

write "2\n" ─────────────► read "2\n"
sleep                      process 2

write "3\n" ─────────────► read "3\n"

The pipeline carries data while both sides are alive.

That is streaming, not completed-string substitution.

The operating system schedules the stages independently. Their execution may overlap in an uneven sequence depending on available data, CPU time, pipe capacity and program buffering. The shell establishes the connections; it does not choreograph every individual read and write.

Pipes carry byte streams

An ordinary pipe provides a unidirectional byte stream:

writer
  │
  │ bytes
  ▼
kernel pipe buffer
  │
  │ bytes
  ▼
reader

The pipe itself does not understand:

lines
JSON objects
filenames
database rows
log records

It only carries bytes.

Any structure comes from an agreement between the programs using it.

For example:

printf 'one\ntwo\nthree\n' | wc -l

works in terms of lines because:

printf writes newline characters;
wc -l interprets newline characters as line boundaries.

The pipe merely transports the bytes.

pipe
  │
  └── byte stream

commands
  │
  └── decide what those bytes mean

Pipes do not preserve application messages

Suppose one process performs two writes:

write "HELLO"
write "WORLD"

The receiving process must not assume that two corresponding reads will return:

read "HELLO"
read "WORLD"

It might observe the stream through reads shaped more like:

read "HEL"
read "LOWOR"
read "LD"

The byte order is preserved, but the pipe does not add application-level message boundaries around separate writes.

writer:

[HELLO] [WORLD]

       │
       ▼

pipe byte stream:

HELLOWORLD

       │
       ▼

reader:

[HEL] [LOWOR] [LD]

Pipes do provide a limited atomicity guarantee for sufficiently small writes: bytes from one such write will not be interleaved with bytes from another writer. On POSIX systems, the relevant limit is PIPE_BUF.

That still does not guarantee that one read() returns exactly one complete write().

If the programs need records, they must define how those records are separated or framed.

Newlines and null bytes are protocol choices

Many command-line pipelines work beautifully because the tools agree that:

one line = one record

For example:

printf '%s\n' \
    'basket created' \
    'item added' \
    'checkout started' |
grep 'checkout'

printf emits newline-delimited records.

grep reads text and applies line-oriented matching.

The pipe connects them.

But newline-delimited data is not safe for everything.

Unix filenames can contain newline characters, so this cannot reliably represent every valid filename:

find . -type f |
while IFS= read -r path; do
    printf 'file: %s\n' "$path"
done

For filenames, null-delimited interfaces are safer:

find . -type f -print0 |
while IFS= read -r -d '' path; do
    printf 'file: %s\n' "$path"
done

The pipe still carries bytes.

The producer and consumer have changed their record-separation agreement from:

newline

to:

null byte

A pipe carries a stream. The processes define the data format.

Streaming, blocking and closure

A pipe can hold only a finite amount of unread data.

If the writer is faster than the reader, data accumulates in the kernel buffer:

fast writer
     │
     ▼
┌───────────────────────┐
│      pipe buffer      │
│  unread unread unread │
└───────────────────────┘
     │
     ▼
slow reader

Once the buffer is full, a normal blocking write waits until the reader consumes enough data to make space.

This creates backpressure:

reader keeps up
    │
    └── writer continues

reader falls behind
    │
    └── pipe fills
          │
          └── writer blocks

The producer cannot necessarily dump unlimited data into memory and disappear. Its progress can become tied to the consumer’s ability to read.

The opposite situation also occurs. If the pipe is empty but at least one process still has its write end open, a blocking reader waits for more data.

reader asks for bytes
       │
       ▼
pipe currently empty
       │
       ├── writer still exists
       │       │
       │       └── wait
       │
       └── no write ends remain open
               │
               └── EOF after buffered data is drained

An empty pipe does not necessarily mean:

The input is finished.

It might mean:

There is no data yet.

EOF means every write end has closed

Suppose:

printf 'hello\n' | cat

cat reads from its standard input.

After printf writes its data and exits, the pipe’s write end is closed.

Once every descriptor referring to that write end has been closed, cat consumes any bytes still buffered in the pipe and then receives end-of-file.

printf writes "hello\n"
        │
        ▼
cat reads "hello\n"
        │
        ▼
printf closes write end
        │
        ▼
no writers remain
        │
        ▼
remaining buffered bytes are consumed
        │
        ▼
cat reads EOF
        │
        ▼
cat exits

This explains why closing unused pipe descriptors matters in lower-level programs.

If some process accidentally keeps a copy of the write end open, the reader may continue waiting because, from the kernel’s perspective, more data could still arrive.

EOF is not a special character travelling through the pipe.

It is a condition created when no write-side descriptors remain open and the buffered data has been consumed.

No readers means `SIGPIPE`

Now reverse the situation.

What happens when the reader exits while the writer is still producing data?

Consider:

yes | head -n 5

yes can produce an effectively endless stream:

y
y
y
y
...

head -n 5 reads five lines and exits.

Once no process has the pipe’s read end open, the next write cannot succeed normally.

A write to a pipe with no readers generates SIGPIPE. If that signal does not terminate the process because it is ignored, blocked or handled, the write also fails with EPIPE.

yes
 │
 │ writes
 ▼
pipe
 │
 ▼
head reads five lines
 │
 ▼
head exits
 │
 ▼
no readers remain
 │
 ▼
yes attempts another write
 │
 ▼
SIGPIPE and failed write

This is useful behaviour. The producer is told:

Nobody is listening anymore.

Without that mechanism, it might continue generating unwanted data forever.

In:

yes | head -n 5

the upstream process being stopped by SIGPIPE is part of the intended operation. That creates an important complication for pipefail: an upstream non-zero status does not always mean the overall task failed.

Failure policy cannot be reduced to one shell option.

Program buffering can hide streaming

Pipes are streaming channels, but programs may also buffer data in user space.

Many C library streams connected to a terminal are line buffered. When the same program’s stdout points to a pipe or ordinary file, it may instead use fuller buffering.

That can produce behaviour such as:

program creates several logical updates
        │
        ▼
updates remain in user-space buffer
        │
        ▼
buffer fills or is flushed
        │
        ▼
larger block enters pipe

There can therefore be at least two buffering layers:

application stream buffer
          │
          ▼
kernel pipe buffer
          │
          ▼
reader

So when pipeline output appears in bursts, that does not necessarily mean the pipe waited for a complete result. The producer itself may have delayed writing its buffered data to the file descriptor.

Concurrent does not mean lockstep.

Pipelines and standard error

An ordinary pipeline connects the first command’s standard output to the next command’s standard input.

Standard error remains separate.

Suppose:

produce_report |
    process_report

The default connections are:

produce_report

stdout ─────► pipe ─────► process_report stdin

stderr ─────────────────► existing stderr destination

That means diagnostics can remain visible on the terminal while ordinary output flows into the next command.

For example:

{
    printf 'report data\n'
    printf 'warning: incomplete source\n' >&2
} |
wc -l

The pipe receives:

report data

while the warning still goes to stderr.

wc -l counts one line.

Bash also supports:

command1 |& command2

This connects both stdout and stderr from the first command to the second command’s stdin. It is shorthand for a form equivalent to:

command1 2>&1 | command2

Conceptually:

command1
 │
 ├── stdout ──┐
 │            ├──► pipe ──► command2 stdin
 └── stderr ──┘

That can be useful when the next command should process both ordinary output and diagnostics.

But combining the channels also destroys the distinction between them. Once both streams enter one byte stream, the consumer cannot automatically know which bytes originally came from stdout and which came from stderr.

In Bash, the implicit 2>&1 performed by |& is applied after explicit redirections belonging to the first command. Mixed uses of 2>... and |& therefore deserve particular care.

Explicit redirection can replace the pipe connection

Bash establishes the pipeline connection before applying a command’s explicit redirections.

Consider:

printf 'hello\n' > output.txt |
wc -l

The pipeline initially connects printf’s stdout to the pipe.

Then:

> output.txt

redirects that stdout to the file instead.

The final arrangement is effectively:

printf stdout ──► output.txt

pipe write end receives no printf output

pipe ──► wc stdin

So wc sees no lines from printf.

The vertical bar does not permanently own stdout. It participates in a sequence of file-descriptor arrangements performed by the shell.

Pipeline stages have separate execution environments

A pipeline is not one command becoming another command.

Its stages execute separately.

In Bash, pipeline elements normally run in subshell environments. The lastpipe shell option can change the final stage’s behaviour when job control is inactive, but relying on that setting can make scripts depend on shell configuration.

This can produce a surprising result:

count=0

printf 'one\ntwo\nthree\n' |
while IFS= read -r line; do
    ((count += 1))
done

printf 'count=%d\n' "$count"

A reader might expect:

count=3

But in ordinary Bash operation, this commonly prints:

count=0

The loop ran in a pipeline subshell.

It updated its own copy of count, not the variable in the parent shell.

parent Bash
 │
 ├── count=0
 │
 └── starts pipeline
        │
        ├── printf stage
        │
        └── while-loop subshell
                 │
                 └── count becomes 3 here

pipeline ends
 │
 ▼
parent still has count=0

The data travelled through the pipe.

Shell state did not travel back with it.

Process substitution can avoid that subshell

In Bash, I can write:

count=0

while IFS= read -r line; do
    ((count += 1))
done < <(printf 'one\ntwo\nthree\n')

printf 'count=%d\n' "$count"

Here the loop runs in the current shell, while the output of printf is provided as redirected input.

The result is:

count=3

Conceptually:

printf stage
      │
      ▼
process-substitution input
      │
      ▼
while loop in current shell
      │
      └── modifies current count

Process substitution is not exactly the same construct as |. Bash may implement it using a named pipe or a /dev/fd-style interface, depending on the system.

The broader lesson is:

A pipeline connects process I/O. It does not merge variables or execution environments.

The pipeline has several statuses

A pipeline may contain several commands:

generate |
validate |
publish

Each command returns its own status.

But the pipeline also needs one combined status that Bash can use in control flow:

if generate | validate | publish; then
    printf 'pipeline succeeded\n'
fi

By default, Bash uses the status of the last command in the pipeline.

So:

false | true
printf 'status=%s\n' "$?"

normally reports:

status=0

The first command returned non-zero.

The last command returned zero.

The default pipeline status is therefore zero, which can hide failures in earlier stages.

Bash’s pipefail option changes that rule:

set -o pipefail

With pipefail enabled, a pipeline returns zero when every command succeeds. Otherwise, it returns the status of the rightmost command that returned non-zero.

set -o pipefail

false | true
printf 'status=%s\n' "$?"

now reports a non-zero status.

This is often useful in automation, but yes | head -n 5 shows why the result still needs interpretation. An intentional SIGPIPE can make an upstream stage look unsuccessful even when the consumer completed its task correctly.

Bash also preserves the individual statuses in the PIPESTATUS array:

false | true | false

statuses=("${PIPESTATUS[@]}")

printf 'first=%s second=%s third=%s\n' \
    "${statuses[0]}" \
    "${statuses[1]}" \
    "${statuses[2]}"

This might produce:

first=1 second=0 third=1

Capturing the array immediately matters because running another command causes Bash to update PIPESTATUS.

The three views are:

default $?
    │
    └── status of final stage

$? with pipefail
    │
    └── status of rightmost unsuccessful stage

PIPESTATUS
    │
    └── status of every stage

The combined status is convenient for control flow.

The individual statuses are useful when I need to understand exactly which stage failed.

`tee` adds another stage

Sometimes I want to keep a copy of the stream while also passing it onwards.

For example:

generate_report |
tee report.txt |
analyse_report

The shape is:

generate_report
       │
       ▼
      pipe
       │
       ▼
      tee
       │
       ├── writes copy to report.txt
       │
       └── writes copy to stdout
                    │
                    ▼
                   pipe
                    │
                    ▼
              analyse_report

tee is not a special branching feature implemented by the pipe.

It is another stage in the pipeline. It reads one byte stream and writes copies to multiple destinations.

commands perform work
pipes connect their streams
Bash arranges the topology

The mental model I’m keeping

The old model:

command1
   │
   ▼
completed output string
   │
   ▼
command2

The new model:

                   BASH
                     │
              creates a pipe
                     │
       ┌─────────────┴─────────────┐
       │                           │
       ▼                           ▼
     STAGE 1                     STAGE 2
       │                           ▲
       │ fd 1                      │ fd 0
       ▼                           │
   write end                    read end
       │                           ▲
       └────── kernel buffer ──────┘

The pipe carries a byte stream.

The stages can run concurrently.

The kernel buffers only a limited amount of unread data.

A reader waits when the pipe is empty but writers remain.

A writer waits when the pipe is full.

After every write-side descriptor closes, the reader drains buffered data and receives EOF.

A write with no readers generates SIGPIPE and fails with EPIPE if the signal does not terminate the process.

The stages keep separate execution environments and separate exit statuses.

And stdout enters the pipe only because Bash connected file descriptor 1 to its write end.

So:

command1 | command2

does not mean:

Run command1, turn everything it prints into a string and substitute that string into command2.

It means:

Create an interprocess communication channel, connect command1’s standard output to command2’s standard input, and run the pipeline.

One character.

Several execution contexts.

A surprising amount of plumbing.

References and further reading

Bash pipelines

GNU Bash Reference Manual: Pipelines Defines Bash pipelines, including the connection between one command’s standard output and the next command’s standard input, |&, subshell execution, waiting behaviour and pipeline exit statuses.

Pipeline execution environments

GNU Bash Reference Manual: Command Execution Environment Explains the subshell environments used for pipeline elements and why variable changes made inside a pipeline command do not normally alter the parent shell.

Pipeline statuses

GNU Bash Reference Manual: Bash Variables Documents PIPESTATUS, the Bash array containing the individual exit statuses from the most recently executed foreground pipeline.

GNU Bash Reference Manual: The Set Builtin Documents the pipefail shell option and its effect on the combined exit status of a pipeline.

Process substitution

GNU Bash Reference Manual: Process Substitution Documents Bash’s <(...) and >(...) syntax, which can expose process input or output through a filename-like interface and avoid some pipeline-subshell patterns.

Linux pipe behaviour

Linux pipe(7) manual page Provides an overview of Linux pipes and FIFOs, including read and write ends, byte-stream semantics, blocking, finite capacity, EOF, SIGPIPE, EPIPE, PIPE_BUF and atomic writes.

Linux pipe(2) manual page Documents the pipe() system call, which creates a unidirectional channel and returns file descriptors for its read and write ends.

Standard-stream buffering

GNU C Library: Buffering Concepts Explains unbuffered, line-buffered and fully buffered streams, including why a program may buffer output differently when connected to a terminal rather than a pipe.

GNU C Library: Standard Streams Describes stdin, stdout and stderr and notes that shell pipes and redirections can determine which files or processes correspond to those streams.

Portable pipe interface

POSIX: pipe() Defines the portable pipe() interface and its creation of separate file descriptors for the read and write ends of a pipe.

What Bash actually connects

The shell creates the connection first

The stages can run at the same time

Pipes carry byte streams

Pipes do not preserve application messages

Newlines and null bytes are protocol choices

Streaming, blocking and closure

EOF means every write end has closed

No readers means SIGPIPE

Program buffering can hide streaming

Pipelines and standard error

Explicit redirection can replace the pipe connection

Pipeline stages have separate execution environments

Process substitution can avoid that subshell

The pipeline has several statuses

tee adds another stage

The mental model I’m keeping

References and further reading

Bash pipelines

Pipeline execution environments

Pipeline statuses

Process substitution

Linux pipe behaviour

Standard-stream buffering

Portable pipe interface

No readers means `SIGPIPE`

`tee` adds another stage