I have been casually saying things like:
“The command printed this.”
That turns out to hide an important question:
Printed it where?
When I run:
printf 'hello\n'
I see:
hello
in my terminal.
It is tempting to imagine that printf somehow knows about my terminal and deliberately writes there.
But then this works:
printf 'hello\n' > hello.txt
and nothing appears on screen.
Instead:
cat hello.txt
shows:
hello
The printf command did not suddenly learn how to write files.
Something else changed.
The missing piece is standard output.
There is one Bash-specific wrinkle in these examples: printf is normally a shell builtin, so Bash does not need to launch a separate external process to run it. Redirection still applies in the same way; Bash arranges the relevant descriptors while the builtin executes.
I am using printf because it makes the I/O easy to see.
Processes usually inherit some conventional channels
Unix-like processes use open file descriptors.
A file descriptor is essentially a small integer a process uses to refer to an open I/O resource.
A program launched in a normal shell environment typically inherits three especially familiar descriptors:
0 standard input
1 standard output
2 standard error
Or, more commonly:
stdin
stdout
stderr
For a C program, the C library exposes the standard streams:
stdin
stdout
stderr
above the underlying file descriptors conventionally numbered:
STDIN_FILENO = 0
STDOUT_FILENO = 1
STDERR_FILENO = 2
The C streams and the file descriptors are related, but they are not literally the same objects. The streams are a library-level abstraction above the descriptors.
So a process launched from an ordinary shell often begins with a small convention already in place:
PROCESS
│
├── fd 0 ──► standard input
├── fd 1 ──► standard output
└── fd 2 ──► standard error
The interesting part is that those arrows do not have to point at the terminal.
stdout is not the terminal
This was the important correction for me.
stdout does not mean:
screen.
It means:
the process’s standard-output channel.
Where that channel currently leads is a separate question.
In an ordinary interactive shell, it might look something like:
process
│
│ fd 1
▼
terminal
That is why:
printf 'hello\n'
appears on screen.
But Bash can arrange this instead:
process
│
│ fd 1
▼
hello.txt
The program can continue writing to:
fd 1
without needing to know that its output now ends up in a file.
That is rather elegant.
The program produces output.
The execution environment decides where that output goes.
Redirection changes the destination
Consider:
printf 'hello\n' > hello.txt
The > syntax belongs to the shell.
Bash interprets it before the command is executed.
For an output redirection such as:
> hello.txt
Bash opens the destination and arranges for the command’s standard output, file descriptor 1, to refer to it.
Conceptually:
before redirection
fd 1 ─────────► terminal
after:
> hello.txt
fd 1 ─────────► hello.txt
Then printf runs.
As far as the command is concerned, it is still writing to standard output.
The plumbing underneath has changed.
This makes redirection feel much less magical.
> and 1> mean the same thing here
Because > defaults to standard output, these are equivalent:
printf 'hello\n' > hello.txt
and:
printf 'hello\n' 1> hello.txt
The second version makes the file descriptor explicit:
1>
│
└── redirect file descriptor 1
I probably would not normally write the 1, but understanding that it is there conceptually makes the rest of shell redirection much easier.
Standard error is a different channel
Now consider a command that fails:
ls /definitely-not-here
I might see:
ls: cannot access '/definitely-not-here': No such file or directory
What happens if I redirect standard output?
ls /definitely-not-here > output.txt
The error message still appears on my terminal.
And:
cat output.txt
is empty.
That initially seems odd until I remember:
stdout = fd 1
stderr = fd 2
I redirected only:
fd 1
The diagnostic was written to:
fd 2
which was still connected to the terminal.
So the process effectively had:
fd 1 ─────────► output.txt
fd 2 ─────────► terminal
Two separate channels.
Redirecting stderr
I can redirect file descriptor 2 explicitly:
ls /definitely-not-here 2> error.txt
Now the terminal stays quiet.
The diagnostic is in:
cat error.txt
So:
2>
│
└── redirect standard error
and:
1>
│
└── redirect standard output
are independent operations.
That means I can separate normal results and diagnostics:
some_command > output.txt 2> errors.txt
giving:
PROCESS
│
├── fd 1 ─────► output.txt
│
└── fd 2 ─────► errors.txt
This is starting to make the names feel literal.
Standard output is where ordinary output goes.
Standard error is a separate conventional destination for diagnostics and errors.
stderr is useful precisely because it is separate
Suppose a command produces machine-readable output:
42
84
126
but also needs to report:
warning: skipped malformed record
If everything goes to standard output:
42
84
warning: skipped malformed record
126
a program consuming that output now has to distinguish data from diagnostics.
Instead:
stdout:
42
84
126
and:
stderr:
warning: skipped malformed record
allow different kinds of information to remain separate.
Conceptually:
COMMAND
│
├── stdout ──► results
│
└── stderr ──► diagnostics
Well-designed command-line tools use that separation deliberately.
The operating system itself does not decide which bytes count as results and which count as diagnostics. I will come back to that distinction below.
This connects to exit status too
Earlier I separated a command’s output from its exit status.
Now the picture is getting richer.
A command can communicate through at least three different channels:
COMMAND
│
┌─────────┼─────────┐
│ │ │
▼ ▼ ▼
stdout stderr exit status
│ │ │
▼ ▼ ▼
results diagnostics outcome
For example:
demo()
{
printf 'result=42\n'
printf 'warning: something odd happened\n' >&2
return 7
}
demo is a Bash function, so like the builtin printf examples above, it does not automatically require a separate external process. The same redirection model still applies.
Running:
demo
printf 'status=%s\n' "$?"
could produce:
result=42
warning: something odd happened
status=7
Three pieces of information.
Three different purposes.
That is much cleaner than treating everything visible on a terminal as one undifferentiated blob called “output”.
What does >&2 mean?
This syntax:
printf 'something went wrong\n' >&2
looked slightly arcane at first.
The important idea is that Bash can duplicate file descriptors.
>&2 says, in effect:
send this command’s standard output to wherever file descriptor 2 currently goes.
So:
printf 'error\n' >&2
lets printf, which normally writes to stdout, send this particular output through the stderr destination instead.
Conceptually:
printf stdout
│
│ >&2
▼
same destination as fd 2
The shell is rearranging file descriptors again.
Sending stdout and stderr to the same place
Suppose I want both channels in one file.
I can write:
some_command > combined.log 2>&1
This becomes much easier to read when I translate it literally.
First:
> combined.log
means:
fd 1 ──► combined.log
Then:
2>&1
means:
make file descriptor 2 refer to the same destination file descriptor 1 currently refers to.
Result:
fd 1 ─────┐
├────► combined.log
fd 2 ─────┘
Bash also has:
some_command &> combined.log
as a convenient way to redirect both standard output and standard error to the same file.
But understanding:
> combined.log 2>&1
is much more useful because it exposes what is happening to the descriptors.
The order matters
This is where shell redirection gets particularly interesting.
Compare:
some_command > output.txt 2>&1
with:
some_command 2>&1 > output.txt
They look almost identical.
They are not.
Bash processes redirections from left to right.
For the first:
> output.txt 2>&1
step one:
fd 1 ──► output.txt
fd 2 ──► terminal
step two:
fd 2 ──► wherever fd 1 points
so the final result is:
fd 1 ─────┐
├────► output.txt
fd 2 ─────┘
Now reverse the order:
2>&1 > output.txt
First:
2>&1
duplicates stdout’s current destination.
At that moment:
fd 1 ──► terminal
fd 2 ──► terminal
Then:
> output.txt
moves only stdout:
fd 1 ──► output.txt
fd 2 ──► terminal
The stderr descriptor does not magically follow stdout afterwards.
That makes 2>&1 feel less like:
forever glue stderr to stdout.
It is closer to:
duplicate where stdout points right now.
That single idea explains a surprising amount of Bash punctuation.
A pipe is another possible destination
Files are not the only thing stdout can point to.
Consider:
printf 'one\ntwo\nthree\n' | wc -l
Here the shell creates a pipe.
The first command’s standard output is connected to the second command’s standard input.
Conceptually:
printf
│
│ stdout
▼
PIPE
│
│ stdin
▼
wc -l
So the output never needs to visit the terminal or a temporary file.
It flows directly between processes.
This finally makes the phrase pipeline feel literal.
Pipes normally carry stdout, not stderr
Suppose:
ls /tmp /definitely-not-here | wc -l
ls may produce:
- ordinary directory entries on stdout;
- an error about the missing path on stderr.
The pipe connects:
ls fd 1
│
▼
pipe
│
▼
wc stdin
But ls’s stderr is still separate:
ls fd 2 ──► terminal
So by default:
stdout ──► pipe
stderr ──► wherever stderr already points
Bash also supports:
command1 |& command2
which connects both standard output and standard error from the first command into the pipe.
Conceptually:
command1
│
├── stdout ──┐
│ ├──► pipe ──► command2 stdin
└── stderr ──┘
Again, the shell is mostly doing plumbing.
The terminal is just one possible endpoint
So far my descriptors have pointed to:
terminal
regular file
pipe
but file descriptors are more general than that.
A process can have descriptors referring to things such as:
regular files
directories
pipes
sockets
devices
This helps explain why Unix programs can often be composed without caring what sits on the other side of their I/O.
Imagine a program that writes:
alpha
beta
gamma
to stdout.
It could be used as:
some_command
and a human sees the output.
Or:
some_command > data.txt
and a file receives it.
Or:
some_command | sort
and another process receives it.
The producer can remain unchanged:
some_command
│
│ stdout
▼
?????????
The question marks might be:
terminal
file
pipe
The program often does not need to care. Its interface remains:
write bytes to file descriptor 1.
The execution environment supplies the destination.
A program can inspect its environment and behave differently when, for example, stdout is attached to a terminal. But that is optional behaviour layered on top of the file-descriptor abstraction.
That decoupling is a large part of what makes small Unix command-line tools so composable.
/proc lets me look at the plumbing
This connects nicely back to my first wander through /proc.
Linux exposes a process’s open file descriptors under:
/proc/<pid>/fd/
For my current shell:
ls -l "/proc/$$/fd"
I might see entries such as:
0 -> /dev/pts/0
1 -> /dev/pts/0
2 -> /dev/pts/0
The exact target depends on my environment.
But conceptually this tells me:
stdin ──► terminal
stdout ──► terminal
stderr ──► terminal
Now try a process whose stdout has been redirected.
For example:
sleep 120 > sleep-output.txt &
pid=$!
ls -l "/proc/$pid/fd"
I can inspect what its descriptors refer to while the process is alive.
The abstract shell syntax:
>
has become visible process state.
That is rather satisfying.
/dev/stdout and friends
On many Unix-like systems I can also encounter:
/dev/stdin
/dev/stdout
/dev/stderr
These provide filesystem-style names associated with the standard descriptors.
On Linux they commonly lead back into the process’s own descriptor view.
So:
/dev/stdout
does not mean:
the global system output device.
It means, effectively:
this process’s standard-output descriptor.
Which again means its eventual destination depends on the execution context.
Programs can inherit the shell’s descriptors
This also explains something from my earlier process notes.
When Bash starts an external command, that command normally inherits an execution environment containing open file descriptors prepared by the shell.
So, very roughly:
SHELL
fd 0 ──► terminal
fd 1 ──► terminal
fd 2 ──► terminal
│
│ execute command
▼
COMMAND
fd 0 ──► terminal
fd 1 ──► terminal
fd 2 ──► terminal
But if Bash performs redirection first:
some_command > output.txt
the child execution environment becomes:
COMMAND
fd 0 ──► terminal
fd 1 ──► output.txt
fd 2 ──► terminal
The program does not need special code for:
running interactively
versus:
being redirected
It can simply use its standard streams.
Redirection is not the same thing as the program writing a file
This distinction is worth keeping.
These two operations can produce a file:
some_command > output.txt
and an application that explicitly does something like:
open("output.txt")
write(...)
But architecturally they are different.
In the first case:
PROGRAM
│
└── writes to stdout
SHELL
│
└── arranged stdout → output.txt
In the second:
PROGRAM
│
├── chooses output.txt
├── opens it
└── writes to it
Shell redirection allows the caller to decide where standard output goes without requiring the program to know a filename.
That is a very useful separation of responsibility.
stdout and stderr are conventions, not meanings enforced by Linux
One caveat.
The operating system knows:
file descriptor 1
file descriptor 2
It does not inspect the prose being written and enforce:
this looks like a normal result
or:
this sounds sufficiently error-like
Applications choose how to use these conventional channels.
So:
stdout = results
stderr = diagnostics
is a useful convention.
It is not a content-classification system built into the kernel.
That is why well-behaved command-line programs matter.
Composition depends partly on them respecting the interface.
The mental model I’m keeping
My old model was:
command
│
▼
prints to terminal
The new model is:
PROCESS
│
┌─────────────┼─────────────┐
│ │ │
▼ ▼ ▼
fd 0 fd 1 fd 2
stdin stdout stderr
│ │ │
▼ ▼ ▼
source destination destination
And the shell can rearrange those connections:
terminal
file
pipe
another process
before the program runs.
So:
command > file
does not mean:
ask the command to save its output to this file.
It means something closer to:
arrange for this command’s standard-output file descriptor to refer to this file, then run the command.
Likewise:
command1 | command2
means:
connect command1’s stdout to command2’s stdin.
Once I think in file descriptors, the punctuation starts becoming plumbing rather than magic.
And that feels like the right place to end my first year wandering around Linux.
I began with:
What exactly is a process?
and somehow ended up staring at file descriptor 2.
Progress, presumably.
References and further reading
Bash redirection
GNU Bash Reference Manual: Redirections Documents Bash’s redirection syntax, including standard input and output file descriptors, output redirection, appending, descriptor duplication and the importance of processing redirections from left to right.
Bash pipelines
GNU Bash Reference Manual: Pipelines
Describes how Bash connects one command’s standard output to another command’s standard input through a pipe, and how |& additionally connects standard error.
Standard streams
GNU C Library: Standard Streams
Introduces the predefined stdin, stdout and stderr streams available to a program and explains their conventional roles as standard input, normal output and diagnostic output.
File descriptors and standard streams
GNU C Library: Descriptors and Streams
Documents the relationship between streams and file descriptors, including STDIN_FILENO, STDOUT_FILENO and STDERR_FILENO, whose values are 0, 1 and 2.
Linux process file descriptors
Linux proc_pid_fd(5) manual page
Documents /proc/<pid>/fd/, where Linux exposes one entry for each open file descriptor belonging to a process. It also identifies descriptors 0, 1 and 2 as standard input, standard output and standard error.
Per-descriptor information
Linux proc_pid_fdinfo(5) manual page
Documents /proc/<pid>/fdinfo/, which exposes additional information about individual open file descriptors.