Shell quoting: one character, several completely different meanings

One character can live several completely different lives in Bash.

Take:

*

It might mean “match filenames”:

printf '%s\n' *.log

It might be multiplication inside an arithmetic expression:

printf '%s\n' "$((6 * 7))"

Or it might be an ordinary asterisk that I want to print:

printf '%s\n' '*'

Likewise, $ can introduce parameter expansion, command substitution or arithmetic expansion. A space can separate arguments or live inside one argument. A semicolon can terminate a command or be literal data.

The character has not changed. Its context has.

That seems to be the central problem shell quoting solves:

Is this character part of the shell language, or is it data that should be passed to a command?

The shell sees the command before the program does

Suppose I run:

printf '%s\n' hello

Bash parses the source and eventually invokes printf with these values:

command name:
    printf

argument 1:
    %s\n

argument 2:
    hello

The quotes around:

'%s\n'

do not become part of the argument. They are shell syntax: Bash uses them while interpreting the command, then removes them before execution.

So printf receives:

%s\n

not:

'%s\n'

A useful first rule is:

Shell quotes control how Bash interprets source text. They are not normally characters passed to the command.

Inspecting the arguments Bash creates

Visible output can hide argument boundaries. This Bash function makes them explicit:

show_args()
{
    printf 'argument count: %d\n' "$#"

    local index=1
    local argument

    for argument in "$@"; do
        printf 'argument %d: <%s>\n' "$index" "$argument"
        ((index += 1))
    done
}

For example:

show_args one two

produces:

argument count: 2
argument 1: <one>
argument 2: <two>

But:

show_args 'one two'

produces:

argument count: 1
argument 1: <one two>

The visible words are the same. The command interface is not.

The angle brackets make ordinary boundaries easy to see, but they do not expose embedded tabs, newlines or carriage returns clearly. For those cases, a shell-escaped view is more useful:

show_args_escaped()
{
    printf 'argument count: %d\n' "$#"

    local index=1
    local argument

    for argument in "$@"; do
        printf 'argument %d: %q\n' "$index" "$argument"
        ((index += 1))
    done
}

%q is a Bash-specific printf conversion. It displays a value in a reusable shell-escaped form rather than printing the value literally.

Unless noted otherwise, the examples below assume Bash’s default IFS, default globbing options and the sample filenames shown.

Single quotes preserve literal text

Single quotes are the strongest ordinary quoting mechanism:

show_args '$HOME' '*.log' 'one; two'

Bash passes three literal arguments:

$HOME
*.log
one; two

It does not expand $HOME, interpret *.log as a filename pattern or treat the semicolon as a command terminator.

A useful model is:

'...'
  │
  └── preserve every enclosed character literally

Single quotes are therefore useful for literal strings, regular expressions, awk programs, format strings and text containing shell punctuation, provided I do not need shell expansion inside the quoted text.

Single quotes prevent expansion

Suppose:

name='Miri'

Then:

printf 'Hello, %s\n' '$name'

prints:

Hello, $name

To expand the variable, I need a different quoting context:

printf 'Hello, %s\n' "$name"

which prints:

Hello, Miri

The difference is small in syntax and enormous in behaviour:

'$name'     literal text
"$name"     expand name while preserving its result

Putting a single quote in literal text

A single quote cannot appear inside an ordinary single-quoted string. This does not mean what it appears to mean:

printf '%s\n' 'Miri's file'

The quote after Miri ends the quoted section. A backslash is not special inside single quotes either, so this is not a solution:

printf '%s\n' 'Miri\'s file'

The usual portable technique is to leave the single-quoted context, add an escaped quote, then re-enter it:

printf '%s\n' 'Miri'\''s file'

Bash joins adjacent quoted and unquoted fragments into one shell word, so the command receives:

Miri's file

The syntax is correct, although visually rather crunchy. Bash’s $'...' form, discussed later, can be clearer when a value contains several single quotes.

Double quotes allow expansion and preserve boundaries

Double quotes still allow parameter expansion, command substitution and arithmetic expansion:

name='Miri'

show_args "Hello, $name"
show_args "$(date +%Y-%m-%d)"
show_args "$((20 + 1))"

For an ordinary scalar expansion, double quotes preserve the result as one argument. This is the common meaning:

Expand this value, but do not split it or interpret wildcard characters inside it as filenames.

There are deliberate exceptions. In particular, "$@" and "${array[@]}" can each produce several separately quoted arguments. They preserve boundaries rather than forcing an entire list into one argument.

The important contrast: $variable and "$variable"

Suppose:

directory='/srv/application logs'

An unquoted expansion:

show_args $directory

is eligible for word splitting and produces two arguments under the stated assumptions:

argument count: 2
argument 1: </srv/application>
argument 2: <logs>

The quoted expansion:

show_args "$directory"

produces one:

argument count: 1
argument 1: </srv/application logs>

Conceptually:

$directory
    │
    ├── parameter expansion
    ├── word splitting may follow
    └── filename expansion may follow

while:

"$directory"
    │
    ├── parameter expansion
    └── one scalar value remains one argument

Quoted expansions should therefore be the default, not merely because a value might contain spaces, but because a scalar variable normally represents one value.

Unquoted expansions can activate filename patterns

Suppose the current directory contains:

application.log
database.log
notes.txt

and:

pattern='*.log'

Then:

show_args $pattern

produces two arguments:

argument count: 2
argument 1: <application.log>
argument 2: <database.log>

The processing is roughly:

parameter expansion
        │
        ▼
      *.log
        │
        ▼
filename expansion
        │
        ▼
application.log database.log

But:

show_args "$pattern"

passes one literal argument:

argument count: 1
argument 1: <*.log>

The variable’s contents are not automatically inert. The context in which I expand them still matters.

Command substitution follows the same rule

Suppose:

output=$(printf 'one two\n')

Then:

show_args $output

produces two arguments, while:

show_args "$output"

produces one argument containing a space.

The practical rule is broader than “quote your variables”:

Quote an expansion when its result should retain its argument boundaries.

That includes common forms such as:

"$variable"
"$(command)"
"${array[index]}"

Empty values show the interface difference

Suppose:

value=''

Then:

show_args "$value"

passes one empty argument:

argument count: 1
argument 1: <>

But:

show_args $value

passes no argument under the stated assumptions:

argument count: 0

An empty argument and a missing argument are not equivalent. Quoting preserves the argument’s existence even when its value contains no characters.

Assignments are a special context

This is safe and idiomatic:

result=$(some_command)

In a simple assignment, Bash does not perform word splitting or filename expansion on the assignment value. The following form is also valid:

result="$(some_command)"

The quotes are optional here, although some people keep them for visual consistency. Once the value is used as a command argument, ordinary quoting rules apply again:

process_result "$result"

Context matters more than a mechanical rule to quote every expansion everywhere.

Backslash also depends on context

Outside quotes, an unquoted backslash preserves the literal value of the next character, except when it forms a line continuation with a newline:

show_args one\ two
show_args \*

These pass one argument containing a space and one literal asterisk.

Inside double quotes, backslash has special behaviour only before a limited group of characters, notably:

$
`
"
\
newline

For example:

show_args "cost: \$5"
show_args "She said \"hello\""

But:

show_args "C:\temp"

does not turn \t into a tab. The backslash remains because t is not special to Bash in that double-quoted context.

Shell escaping should therefore not be confused with the rules of C strings, JSON, regular expressions or printf format strings. A backslash may be interpreted by Bash, by the called program, by both or by neither.

A backslash immediately followed by a newline continues a command across physical lines:

printf '%s\n' \
    'one' \
    'two' \
    'three'

The backslash must be the final character before the newline. A trailing space means it quotes that space instead, creating a tiny invisible failure with the personality of a loose floorboard.

A shell word can use several quoting styles

Bash can construct one argument from adjacent fragments:

name='Miri'

show_args 'Hello, '"$name"'!'

The shell source contains three pieces:

'Hello, '    "$name"    '!'

Each uses the quoting rule it needs. Because there is no unquoted separator between them, Bash joins them into one word:

Hello, Miri!

An argument does not need to use one quoting style from beginning to end.

Parsing happens before expansion

Bash first recognises shell syntax in the source it is reading. Later, it performs expansions. Text produced by an ordinary expansion is not normally parsed again as complete shell source.

For example:

value='one; printf hacked'
show_args $value

The unquoted expansion is eligible for word splitting, so it may become three arguments:

one;
printf
hacked

However, the semicolon that came from the variable does not become a command terminator. It was not present as shell syntax when Bash parsed the original command.

Likewise, quote characters stored in a variable do not open a new quoting context:

value='"one two"'
show_args $value

The quote characters are data. They do not protect the embedded space from word splitting.

This distinction explains why these are different:

show_args one\ two
show_args $value

In the first command, the backslash protects the space while Bash tokenises the original source. In the second, text arrives later through an unquoted expansion.

Expansion can trigger word splitting and filename expansion. It does not usually create new operators, quoting syntax or commands.

eval is unusual and dangerous precisely because it performs another full parsing pass.

"$@" preserves an argument list

Suppose a script receives:

./script 'one two' '*.log'

The caller supplied two arguments. Inside the script:

show_args "$@"

preserves them as two separately quoted words. Conceptually:

"$@"  →  "$1" "$2" "$3" ...

By contrast:

show_args $@

allows the values to undergo further splitting and filename expansion, while:

show_args "$*"

joins all positional parameters into one word using the first character of IFS.

For forwarding an argument list, the normal pattern is:

some_command "$@"

The same boundary-preserving idea applies to arrays:

some_command "${arguments[@]}"

These forms are the important exceptions to the shorthand claim that double quotes always produce one argument. They can produce several arguments, but each original element keeps its own boundary.

Special quoting contexts

Quoting patterns in [[ ... ]]

Bash’s [[ ... ]] syntax has its own rules:

filename='application.log'

if [[ $filename == *.log ]]; then
    printf 'log file\n'
fi

The unquoted right-hand side is a pattern. Quoting it changes the comparison:

if [[ $filename == "*.log" ]]; then
    printf 'log file\n'
fi

Now Bash compares against the literal text *.log.

The expansion on the left does not need ordinary double quotes here because [[ ... ]] suppresses word splitting and filename expansion for it. This is a special Bash syntactic context, not a reason to stop quoting ordinary command arguments.

“Always quote everything” is therefore too blunt. Quoting is normally the safe default for expansions, but sometimes I deliberately leave shell syntax unquoted because I want Bash to interpret it as syntax.

Quoting a here-document delimiter

With an unquoted delimiter:

name='Miri'

cat <<EOF
Hello, $name
EOF

Bash expands $name.

With a quoted delimiter:

cat <<'EOF'
Hello, $name
EOF

the body is not subjected to parameter expansion, command substitution or arithmetic expansion. The output contains the literal text:

Hello, $name

The quotes around the delimiter select the expansion behaviour of the entire here-document body.

ANSI-C quoting

Bash also supports ANSI-C-style quoting:

printf '%s' $'one\ntwo\n'

Inside $'...', selected backslash sequences are decoded:

\n    newline
\t    tab
\\    backslash
\'    single quote

So:

show_args $'Miri\'s file'

passes one argument containing:

Miri's file

This is a Bash feature rather than part of the simplest portable shell-quoting toolkit. Ordinary single quotes, double quotes and backslashes are often enough.

printf is a better microscope than echo

Quoting demonstrations often use:

echo "$value"

That can be misleading because echo implementations may treat options and backslash sequences differently.

For deliberate output, especially in scripts, printf has a clearer interface:

printf '%s\n' "$value"

The format string is syntax supplied by the script. The value is data supplied as a separate argument.

Prefer that form to:

printf "$value"

If value contains format directives, the second command gives user-controlled text a new meaning.

There it is again:

Is this text syntax, or is it data?

Quoting does not make eval safe

Suppose I build a command in one string:

command_text='printf "%s\n" "$HOME"'

and execute:

eval "$command_text"

The quotes protect the value while it is passed to eval. They do not stop eval from parsing that value as shell source.

This extra parsing pass allows characters that were data to become operators, expansions, redirections or commands. It is especially dangerous when any part of the string came from an untrusted source.

A safer design is usually to represent a command as an array:

command=(
    printf
    '%s\n'
    "$HOME"
)

"${command[@]}"

The command name and argument boundaries remain explicit. Nothing is reconstructed into shell source and reparsed.

A string is not an argument list, and quoting cannot reliably turn arbitrary data into safe shell program text.

The rule and mental model I am keeping

My first shell-quoting rule was:

Put quotes around paths with spaces.

That is too narrow. Spaces are only one symptom.

A better rule is:

Quote expansions when their results should retain their value or argument boundaries.

These forms should feel normal:

cd "$directory"
rm -- "$path"
printf '%s\n' "$message"
some_command "$@"
process_result "$result"

An unquoted expansion should make me pause:

some_command $value

Perhaps I genuinely want word splitting or filename expansion. That should be a deliberate decision rather than an accidental side effect.

The compact model is:

UNQUOTED EXPANSION

$variable
    │
    ├── expand
    ├── possibly split
    └── possibly expand filenames


SINGLE QUOTED

'$variable'
    │
    └── literal characters


DOUBLE-QUOTED SCALAR EXPANSION

"$variable"
    │
    ├── expand
    └── preserve one argument


BOUNDARY-PRESERVING LIST EXPANSION

"$@"
"${array[@]}"
    │
    └── preserve each element as its own argument


BACKSLASH

\*
    │
    └── preserve the next character literally
        in this context

Bash source begins as characters. Quoting helps determine which characters remain shell language and which become literal data. It controls token boundaries, expansion, splitting, filename matching, pattern matching and here-document behaviour.

One character can therefore be syntax, a pattern, a separator, an operator or literal data depending on where it appears and how it is quoted.

The shell is not being inconsistent. It is interpreting a language whose punctuation changes meaning by context.

My job is to make that context explicit enough that the shell, the called command, the next reader and future me all agree about what the characters are supposed to mean.

That is a lot of responsibility for two tiny quotation marks.

References and further reading

Bash quoting rules

GNU Bash Reference Manual: Quoting Introduces Bash’s quoting mechanisms and explains that quoting removes the special meaning of characters or words to the shell.

GNU Bash Reference Manual: Escape Character Documents the unquoted backslash and its use in preserving the literal value of the following character or continuing a command across lines.

GNU Bash Reference Manual: Single Quotes Explains that ordinary single quotes preserve every enclosed character literally and cannot themselves appear inside a single-quoted string.

GNU Bash Reference Manual: Double Quotes Documents which characters retain special meaning inside double quotes and how backslash behaves in that context.

GNU Bash Reference Manual: ANSI-C Quoting Defines Bash’s $'...' form and the backslash escape sequences it decodes.

Shell processing and expansion

GNU Bash Reference Manual: Shell Operation Summarises how Bash reads, tokenises, parses, expands and redirects a command before executing it.

GNU Bash Reference Manual: Shell Expansions Documents Bash’s expansion stages, including parameter expansion, command substitution, word splitting, filename expansion and quote removal.

GNU Bash Reference Manual: Word Splitting Explains how Bash splits the results of unquoted parameter, command and arithmetic expansions using IFS.

GNU Bash Reference Manual: Filename Expansion Documents how unquoted *, ? and [ characters form filename-matching patterns after word splitting.

Positional parameters

GNU Bash Reference Manual: Special Parameters Explains the different behaviours of $@, "$@", $* and "$*", including why "$@" preserves positional parameters as separate arguments.

Conditional patterns

GNU Bash Reference Manual: Bash Conditional Expressions Documents conditional expressions and the pattern-matching behaviour of == and != inside [[ ... ]].

Here-documents

GNU Bash Reference Manual: Here Documents Explains how quoting a here-document delimiter disables expansion within the document body.

Portable shell language

POSIX.1-2017: Shell Command Language Defines the portable shell language, including escaping, single quotes, double quotes, parameter expansion and field splitting.