Cloud Sandbox
All writing

Building a deliberate Bash systems script

Taking a useful Bash command and adding explicit interfaces, validation, failure handling, cleanup, idempotence and operational safety.

about 19 minutes min read

In the previous note, I stopped treating Bash as merely somewhere to type commands.

It is a language for coordinating commands, streams, statuses, files and operating-system state.

That makes even a small script a program.

It also means a useful command can become operationally interesting surprisingly quickly.

Consider the large-log search from the previous note:

#!/usr/bin/env bash

log_dir=${1:-/var/log}

find "$log_dir" -type f -size +100M -printf '%s\t%p\n' |
    sort -nr -k1,1

It is already useful.

But it leaves several questions unanswered:

What if the arguments are invalid?

What if a required command is missing?

What if filesystem traversal fails halfway through?

Where should report data and diagnostics go?

What temporary state will the script leave behind?

What happens when the script runs twice?

What happens when it is interrupted?

This note is about making those answers explicit.

The examples assume Bash and the GNU command-line tools commonly found on Linux. Options such as GNU find -printf and stat --format differ on some other Unix-like systems.

Failure needs more than a shell option

A common script header is:

set -e

or:

set -euo pipefail

The intention is understandable:

Stop when something goes wrong.

But Bash’s errexit behaviour has context-sensitive exceptions.

A non-zero status may be treated differently when the command appears inside:

an if condition
a while or until test
parts of && or || lists
most pipeline positions
a negated command

That does not make set -e useless.

It does mean:

“Strict mode” is not a substitute for explicitly designing how the script handles failure.

For learning scripts, I currently prefer to make important failures visible:

if ! create_report; then
    printf 'report creation failed\n' >&2
    exit 1
fi

or:

create_report ||
    die 'report creation failed'

The behaviour is obvious from the code.

Later, I can decide whether errexit improves a particular script once I understand its rules and exceptions.

Functions give operations names

Once a script grows, functions let me group commands behind meaningful names.

require_command()
{
    local command_name=$1

    if ! command -v "$command_name" >/dev/null 2>&1; then
        printf 'required command not found: %s\n' "$command_name" >&2
        return 1
    fi
}

I can then write:

require_command find
require_command stat
require_command sort

instead of repeating the implementation.

The function name describes the intention:

require_command

rather than forcing the reader to decode:

command -v ... >/dev/null 2>&1

Functions also create useful boundaries inside a script:

validate_input
collect_data
write_report
cleanup
main

This is not object-oriented architecture.

It is simply enough structure to stop the script becoming one uninterrupted kilometre of shell commands.

Functions return statuses

A Bash function behaves like a command.

It returns an exit status.

By default, its status is the status of the last command executed in the function.

I can also return one explicitly:

directory_is_readable()
{
    local directory=$1

    [[ -d $directory && -r $directory ]]
}

Then use it as a condition:

if directory_is_readable "$log_dir"; then
    printf 'scanning %s\n' "$log_dir"
else
    printf 'cannot read directory: %s\n' "$log_dir" >&2
    exit 1
fi

The function does not have to print:

true

or:

false

Its status carries the answer.

This fits the wider command model:

function
   │
   ├── stdout
   ├── stderr
   └── exit status

A shell function can therefore participate in pipelines, conditions and command lists through the same stdout, stderr and status interfaces as an external command.

There is one shell-state caveat: a function placed in a multi-command pipeline will normally run in a subshell environment. Changes it makes to variables or the current directory may therefore disappear when that pipeline component ends.

The command interface composes cleanly.

Mutable shell state needs more care.

Standard streams create interfaces

The distinction between stdout and stderr becomes particularly useful in scripts.

Suppose the script’s output is intended for another program:

./large-logs /var/log |
    upload-report

Then stdout should contain the report data.

Progress messages and warnings should go to stderr:

printf 'scanning %s\n' "$log_dir" >&2

This keeps the channels separate:

SCRIPT
 │
 ├── stdout ──► report data
 │
 └── stderr ──► diagnostics

The caller can then decide what to do with each:

./large-logs /var/log \
    > report.txt \
    2> scan.log

A well-behaved script is easier to use interactively, in another script, from a scheduler or inside a CI job.

The same program can participate in several environments because it exposes conventional process interfaces.

Variables hold shell state

Bash variables let a script remember information between operations:

log_dir=${1:-/var/log}
threshold_mb=${2:-100}
report_file=${REPORT_FILE:-large-files.txt}

They can represent:

configuration
paths
command results
counters
temporary state

But Bash variables do not behave like strongly typed variables in many general-purpose languages.

Unless a special attribute or arithmetic context says otherwise, I should largely think of them as strings interpreted by the shell.

That means careful validation matters.

For example:

if [[ ! $threshold_mb =~ ^[0-9]+$ ]]; then
    printf 'threshold must be a non-negative integer\n' >&2
    exit 2
fi

The script should not blindly assume that:

$2

contains the kind of value it expects.

Inputs crossing a boundary should be treated as untrusted until validated.

That remains true even when the boundary is “Miri typing a command after lunch”.

Arrays preserve argument boundaries

Suppose I want to build a command conditionally.

This is fragile:

options='-type f -name *.log'
find "$log_dir" $options

The variable contains text that I hope Bash later divides into the arguments I intended.

That forces me to reason about another round of splitting and pathname expansion.

A Bash array is usually a better representation:

find_options=(
    -type f
    -name '*.log'
)

find "$log_dir" "${find_options[@]}"

Here each array element remains one argument.

element 1  -type
element 2  f
element 3  -name
element 4  *.log

The quoted expansion:

"${find_options[@]}"

preserves those boundaries.

This is a broader scripting principle:

Represent a list of arguments as an array, not as one string containing shell syntax.

A command is not one textual sentence.

It is a command name plus an ordered collection of arguments.

Arrays model that reality more accurately.

Temporary state needs cleanup

Systems scripts often need temporary files or directories.

A dangerous first attempt might be:

temporary_file=/tmp/report.txt

That path could:

  • already exist;
  • collide with another invocation;
  • belong to another user;
  • be replaced unexpectedly;
  • remain behind after failure.

mktemp creates a unique temporary location more safely:

workspace=$(mktemp -d)

But I still need to remove it.

Bash’s trap builtin lets a script arrange commands to run when particular signals or shell events occur.

cleanup()
{
    if [[ -n ${workspace:-} && -d $workspace ]]; then
        rm -rf -- "$workspace"
    fi
}

trap cleanup EXIT

Now the cleanup function runs as the shell exits.

Conceptually:

script starts
    │
    ▼
create temporary workspace
    │
    ▼
perform work
    │
    ├── success
    ├── explicit failure
    └── interrupted exit
           │
           ▼
        cleanup

This is another place where Bash touches systems concerns directly.

The script is responsible not only for its main operation but also for the state it leaves behind.

Cleanup deserves defensive code

This line should make any systems engineer pause:

rm -rf -- "$workspace"

It is reasonable only because the script controls how workspace is created and checks that it contains a non-empty directory path.

A careless version such as:

rm -rf $workspace

combines several hazards:

  • an unquoted expansion;
  • a possibly empty value;
  • values beginning with -;
  • unintended splitting or pathname expansion.

The safer form uses:

"${workspace}"

to preserve the value and:

--

to mark the end of command options.

Shell scripting often looks lightweight because the syntax is small.

The consequences are still those of operating-system commands.

Idempotence matters in automation

Suppose a script creates a directory:

mkdir /srv/reports

The first run may succeed.

The second may fail because the directory already exists.

If the desired state is simply:

/srv/reports should exist,

then:

mkdir -p /srv/reports

may express the intention better.

Repeated execution converges on the same state.

That is the beginning of idempotent automation.

Compare:

imperative step

"create this directory now"

with:

desired condition

"ensure this directory exists"

Not every shell operation can be idempotent.

Sending an email, charging a card or appending a line may produce a new effect on every run.

But I should always ask:

What happens if this script runs twice?

What happens if the first run stops halfway?

Can the operation safely resume?

How will it distinguish existing desired state
from partial or conflicting state?

A systems script lives in a changing environment.

“Works once on my machine” is a demonstration, not an operational design.

Observation before mutation

Bash makes powerful system changes extremely easy to express.

find "$directory" -type f -delete

That is convenient.

It is also one typo away from a memorable afternoon.

A safer workflow often separates:

discover
validate
display
change
verify

For example:

find "$directory" -type f -print

first.

Then, once the selection is clearly correct, introduce the action.

For reusable tools, a dry-run mode can make that distinction explicit:

if [[ $dry_run == true ]]; then
    printf 'would remove: %s\n' "$path"
else
    rm -- "$path"
fi

The script should make dangerous behaviour harder to trigger accidentally and easier to inspect beforehand.

Bash does not provide that safety automatically.

The script’s design has to supply it.

Environment is another input

A process can inherit environment variables from its parent.

A Bash script may therefore receive configuration from values such as:

PATH
HOME
TMPDIR
LANG
REPORT_FILE

That is useful, but it also means the script’s behaviour may depend on state that is not visible in its arguments.

For example, command lookup depends on PATH.

command -v sort

may find different executables in different environments.

Locale can affect text processing and sorting.

A script that behaves one way interactively may behave differently when run by:

cron
a service manager
a CI runner
another user

because the environment changed.

So robust systems scripts should be deliberate about the environment they require.

That might mean:

  • validating required commands;
  • setting or documenting locale assumptions;
  • avoiding dependence on interactive shell configuration;
  • using explicit paths where appropriate;
  • providing configuration through named variables or arguments.

The environment is part of the program’s input whether I acknowledge it or not.

A more deliberate script

Bringing these ideas together, the large-file report can become something like this:

#!/usr/bin/env bash

set -u
set -o pipefail

readonly program_name=${0##*/}
readonly default_threshold_mb=100

workspace=''

usage()
{
    printf 'Usage: %s [directory] [threshold-megabytes]\n' \
        "$program_name"
}

die()
{
    printf '%s: %s\n' "$program_name" "$*" >&2
    exit 1
}

require_command()
{
    local command_name=$1

    command -v "$command_name" >/dev/null 2>&1 ||
        die "required command not found: $command_name"
}

cleanup()
{
    if [[ -n ${workspace:-} && -d $workspace ]]; then
        rm -rf -- "$workspace"
    fi
}

interrupted()
{
    printf '%s: interrupted\n' "$program_name" >&2
    exit 1
}

trap cleanup EXIT
trap interrupted HUP INT TERM

main()
{
    local log_dir=${1:-/var/log}
    local threshold_mb=${2:-$default_threshold_mb}
    local threshold_bytes
    local paths
    local report
    local path
    local size
    local warning_count=0

    if (( $# > 2 )); then
        usage >&2
        return 2
    fi

    [[ -d $log_dir ]] ||
        die "directory does not exist: $log_dir"

    [[ -r $log_dir ]] ||
        die "directory is not readable: $log_dir"

    [[ $threshold_mb =~ ^[0-9]+$ ]] ||
        die 'threshold must be a non-negative integer'

    require_command find
    require_command stat
    require_command sort
    require_command awk
    require_command mktemp

    threshold_bytes=$((threshold_mb * 1024 * 1024))

    workspace=$(mktemp -d) ||
        die 'could not create temporary workspace'

    paths=$workspace/paths.bin
    report=$workspace/large-files.tsv
    : > "$report"

    printf 'Scanning %s for files of at least %s MiB\n' \
        "$log_dir" "$threshold_mb" >&2

    if ! find "$log_dir" -type f -print0 > "$paths"; then
        die "could not enumerate files beneath: $log_dir"
    fi

    while IFS= read -r -d '' path; do
        if ! size=$(stat --format='%s' -- "$path"); then
            printf 'warning: could not inspect %s\n' "$path" >&2
            ((warning_count += 1))
            continue
        fi

        if (( size >= threshold_bytes )); then
            printf '%s\t%s\n' "$size" "$path" >> "$report"
        fi
    done < "$paths"

    if [[ ! -s $report ]]; then
        printf 'No files met the threshold.\n'
    else
        sort -nr -k1,1 "$report" |
            awk -F '\t' '
                {
                    printf "%10.2f MiB  %s\n", $1 / 1048576, $2
                }
            '
    fi

    if (( warning_count > 0 )); then
        printf 'Completed with %d inspection warning(s).\n' \
            "$warning_count" >&2
        return 1
    fi
}

main "$@"

This is still a small script.

It is also doing quite a lot.

arguments
   │
   ▼
validation
   │
   ▼
dependency checks
   │
   ▼
temporary workspace
   │
   ▼
checked filesystem traversal
   │
   ▼
per-file inspection
   │
   ▼
selection
   │
   ▼
sorting and formatting
   │
   ▼
stdout report
stderr diagnostics
exit status
   │
   ▼
cleanup

Bash is coordinating commands, shell state, process interfaces and failure behaviour.

That is systems programming, even if it is not implementing a kernel or allocating memory manually.

A limitation in this example

The file enumeration and scanning loop use null-delimited filenames, so spaces and ordinary shell metacharacters do not break the input.

The enumeration step is also checked before processing begins, so a failing find cannot quietly disappear behind process substitution.

The temporary report itself is tab and newline delimited.

That means the reporting format assumes the selected log paths do not contain tabs or newlines.

For conventional system log paths, that may be an acceptable documented constraint.

For a tool intended to process arbitrary hostile filenames, I would need a representation that preserves those names through the complete sorting and reporting pipeline.

The important lesson is not to claim universal safety where the data format does not provide it.

The mental model I’m keeping

A deliberate systems script has more than a successful main path.

It has an interface.

It validates the environment it depends on.

It separates report data from diagnostics.

It gives failures meanings rather than hoping one shell option understands them all.

It owns the temporary state it creates.

It considers interruption, repeated execution and partial completion.

So the progression is no longer:

useful command
      │
      ▼
put it in a file
      │
      ▼
finished

It is closer to:

useful operation
      │
      ▼
define inputs and outputs
      │
      ▼
validate assumptions
      │
      ▼
make failure visible
      │
      ▼
control temporary and persistent state
      │
      ▼
verify and clean up
      │
      ▼
return a meaningful status

Bash still has an appropriate scale.

It is strongest when commands and operating-system interfaces remain the main abstractions.

But within that scale, “just a script” is not an engineering exemption.

The machine will execute the commands exactly as seriously as commands written by a much larger program.

I should design the script accordingly.

References and further reading

Bash language and functions

GNU Bash Reference Manual The primary Bash language reference.

GNU Bash Reference Manual: Shell Functions Explains how Bash functions group commands, execute in the current shell context and return the status of their final command unless another status is supplied.

Failure, pipelines and redirection

GNU Bash Reference Manual: Pipelines Defines pipeline status, pipefail and the execution environment used for pipeline components.

GNU Bash Reference Manual: Redirections Documents how Bash opens, closes, duplicates and rearranges file descriptors before commands execute.

GNU Bash Reference Manual: Process Substitution Documents Bash’s process-substitution mechanism. Commands inside process substitution run asynchronously, which is why their failures need deliberate handling.

GNU Bash Reference Manual: Exit Status Documents Bash’s exit-status model.

Traps and cleanup

GNU Bash Reference Manual: Bourne Shell Builtins Includes the reference for trap, which lets scripts arrange actions for signals and shell events such as EXIT.

Safe filename handling

GNU Findutils: Safe File Name Handling Explains null-delimited filename handling with tools such as find -print0.

Command-line utilities

GNU Coreutils Manual Documents many of the utilities coordinated by the complete example.

Static analysis

ShellCheck A static-analysis tool for shell scripts that identifies common quoting, expansion, portability and control-flow problems.