Bash & Shell

Bash and the Unix tools with step-through labs (expansion, redirections, pipelines, set -e, subshells, awk), production workflows, 11 runnable problems and the GNU vs macOS flags.

Overview

This is a working reference for Bash and the standard Unix tools, built around what the shell actually does with a line you type. Each mechanism has a step-through lab you can edit and re-trace, the tables stay for quick lookup, and the page ends with production workflows and eleven problems you run here. It assumes Bash 5 on Linux with the GNU tools; where macOS or BSD differs, the tables say so.

> [!IMPORTANT] macOS ships zsh as the default shell (since 10.15) and /bin/bash 3.2, which has no associative arrays, mapfile or ${var^^}. Install a current Bash and start scripts with #!/usr/bin/env bash.

> [!NOTE] Runnable blocks (bash-run) run on bash 5.2 with GNU coreutils, sed, grep and findutils, and mawk as awk, in an empty temporary directory with no network. Every one makes its own input and prints ok: or FAIL.

What happens when you press Enter

Read — readline edits the line; Enter submits it — a here-doc or open quote makes bash keep reading (> prompt)
Parse — splits into words and operators: pipelines, lists, redirections — x = 1 is a command named x
Expand — brace, tilde, $var, $(( )), $( ), splitting, globbing, quote removal — unquoted $var splits and globs
Redirect — in the child, applies , 2>&1 left to right — order of 2>&1 matters
Execute — builtins and functions run in this shell; programs are found on PATH, forked, exec'd — cd in a pipeline or ( ) does not change your shell
Wait — collects every exit status into $? and PIPESTATUS — only the last stage counts unless pipefail

Start every script the same way

Bash keeps going after a failed command, and an unset variable quietly becomes an empty string. These lines change that, and guarantee cleanup.

set -e — exit when a command fails — many exceptions: see the lab below
set -u — an unset variable is an error — "$@" may still be empty; use ${var:-} where unset is fine
set -o pipefail — a pipeline fails if any stage fails — off by default; … \ — head can now fail with 141
set -E — functions and subshells inherit the ERR trap — without it, an ERR trap set at the top misses failures inside functions
trap '…' EXIT — cleanup on every exit path — cannot run after kill -9
local v=$(cmd) — hides a failure from set -e — local returns 0; declare first, then assign

Expansion: the order behind every quoting bug

Edit the command line, the variables or the file list, then trace it. Each box is one word; the last frame is exactly the argv the program receives.

Word splitting and globbing act only on the *results* of unquoted expansions, and they come after everything else. Double quotes switch off both, which is why "$var" is one word and $var can be many.

"text $var" — yes — almost everything
'text $var' — no, fully literal — regexes, awk and sed programs
$'a\tb\n' — no; reads \t, \n, \x41 — tabs and newlines in a string
\$ — escapes one character — a literal $ inside double quotes
unquoted $var — yes, then splits and globs — only when you want splitting

rm $file — rm my report.txt — two arguments: wrong files
rm "$file" — one argument — correct
[ -f $file ] — [ -f my report.txt ] — error: too many arguments
[[ -f $file ]] — no splitting inside [[ ]] — works, but quote anyway

Quoting failures that reached production

rm -rf "$STEAMROOT/"* — a 2015 Steam for Linux script ran with STEAMROOT empty, so the glob became /* and deleted users' files — "${STEAMROOT:?}"/* stops when it is empty or unset
for f in $(ls) — names with spaces split into pieces; a * in a name globs — for f in * or find … -print0
[ -n $token ] — with token empty this is [ -n ], a one-argument test, which is true — [ -n "$token" ] or [[ -n $token ]]
[ $env = prod ] — with env empty: [: =: unary operator expected — [ "$env" = prod ]
cp $src $dst — a path with a space becomes extra arguments — cp -- "$src" "$dst"
args="--out $f"; cmd $args — the string re-splits, $f with a space breaks — an array: args=(--out "$f"); cmd "${args[@]}"

Special parameters

$0 — name the script was run as
$1 to $9, ${10} — positional arguments; $10 means $1 then a 0
$# — number of arguments
"$@" — every argument as its own word; use this to pass arguments on
"$*" — all arguments joined into one word by the first character of IFS
$? — exit status of the last command
$$ — process ID of this shell (the parent's, even inside a subshell)
$! — process ID of the last background job
$_ — last argument of the previous command

Parameter expansion

These run inside the shell, so they are much cheaper than calling basename, dirname or sed in a loop. The examples use path=/tmp/archive/report.tar.gz and var='hello world'.

${path##*/} — report.tar.gz — remove the longest match from the front (basename)
${path%/*} — /tmp/archive — remove the shortest match from the end (dirname)
${path##*.} — gz — extension
${path%.*} — /tmp/archive/report.tar — drop the last extension
${path%%.*} — /tmp/archive/report — drop from the first dot
${#var} — 11 — length
${var:6} and ${var:0:5} — world and hello — substring by offset and length
${var: -5} — world — last 5 characters; the space is required
${var/o/0} and ${var//o/0} — hell0 world and hell0 w0rld — replace first, replace all
${var/#hello/hi} — hi world — replace only at the start (/% for the end)
${var^^}, ${var,,}, ${var^} — HELLO WORLD, hello world, Hello world — change case (Bash 4.0 and later)
${!name} — value of the variable whose name is in name — indirection

The colon forms treat an empty value like an unset one. Without the colon, only unset counts.

${v:-d} — d — d — x
${v-d} — d — empty — x
${v:=d} — d, and assigns it — d, and assigns it — x
${v:+d} — empty — empty — d
${v:?msg} — error, script exits — error, script exits — x

Arithmetic, substitution and sequences

$(( 7 / 2 )) and $(( -7 / 2 )) — 3 and -3 — integers only; division truncates toward zero
$(( 7 % 2 )), $(( 2 ** 10 )) — 1, 1024 — remainder, power
$(( 16#ff )), $(( 010 )) — 255, 8 — a leading 0 means octal
m=08; $(( m + 1 )) — error: value too great for base — months and hours from date; write $(( 10#$m + 1 ))
(( i++ )), (( a > b )) — exit status only — no $ needed on names inside
(( 0 )), and (( i++ )) when i is 0 — status 1 — a zero result is false, so under set -e (( i++ )) ends the script; use i=$((i + 1))
$(cmd) — output of cmd — trailing newlines are removed; nests cleanly
{1..5}, {0..20..5}, {01..03}, {a..e} — 1 2 3 4 5, 0 5 10 15 20, 01 02 03, a b c d e — literals only: brace expansion runs before $n is expanded, so {1..$n} stays as it is
for (( i = 0; i "$b" ] — > is a redirection: it creates a file named by $b, and the test is true — [[ $a > $b ]] or [ "$a" \> "$b" ]
[[ 10 &1 copies whatever fd 1 points at *at that moment*. In >out 2>&1, fd 1 already points at the file, so both land there; in 2>&1 >out, fd 2 copied the terminal (or the pipe) first. Two separate > opens of one file are two independent offsets, so >f 2>f overwrites itself.

> f, >> f — stdout to a file: truncate, append
2> f — stderr to a file
> f 2>&1 — both to the file (POSIX)
&> f, &>> f — both, Bash only (&>> needs Bash 4.0 or later)
2>&1 > f — stdout to the file, stderr to the old stdout
cmd \ — & next — Bash: 2>&1 \ — , applied after cmd's own redirections
3>&1 1>&2 2>&3 3>&- — swap stdout and stderr
/dev/null 2>&1 — discard all output
cmd \ — tee f, tee -a f — show and save; append
echo x \ — sudo tee /etc/f — write a root-owned file: sudo echo x > /etc/f fails because your shell opens the file
exec 3> f ... exec 3>&- — open and close your own descriptor 3
exec > >(tee -a log) 2>&1 — copy the rest of the script's output to a log

[!WARNING] With pipefail, producer — head -n 1 can fail with 141 (128 + SIGPIPE) when the producer still had output after head exited. Under set -e that ends the script. Use a command that stops by itself (grep -m1 pat file), read from head -n 1 log — group in this shell — needs spaces and the final ;
( cd dir && a ) — in a subshell — the cd does not affect the caller

trap cleanup EXIT — any exit, normal or error — delete temp files, release locks
trap '...' ERR — a failing command (same exceptions as set -e) — print $LINENO and $BASH_COMMAND; add set -E for functions
trap '...' INT TERM — Ctrl-C, kill — stop cleanly, then exit 130 or exit 143
trap '' HUP — hangup — ignore it
trap - EXIT — remove a trap

KILL and STOP cannot be caught, blocked or ignored, so no trap runs after kill -9 or an OOM kill.

grep

grep -i pat f — ignore case — both
grep -v pat f — lines that do not match — both
grep -n, -c — line numbers; count of matching lines — both
grep -l, -L — names of files that match; that do not — both
grep -o — print only the matching part — both
grep -w, -x — whole word; whole line — both
grep -q — no output; status only, stops at the first match — both
grep -m1 pat — stop after the first match — both
grep -F 'a.b' — fixed string, no regex — both
grep -E 'a+\ — b' — extended regex (+ ? \ — ( ) { } with no backslash) — both
grep -P '\d+(?=px)' — Perl regex: \d, lookaround — GNU only, and only if built with PCRE; macOS grep rejects -P
grep -rn --include='*.py' pat . — recursive, by file pattern — both
grep -r --exclude-dir=.git pat . — skip a directory — both
grep -A3 -B2, -C2 — lines after, before, around — both
grep -e a -e b, grep -f pats.txt — several patterns — both
grep --line-buffered — flush each line, for tail -f \ — grep \ — … — both

sed

sed 's/a/b/' f — replace the first match on each line — both
sed 's/a/b/g', 's/a/b/2' — all matches; only the second — both
sed 's/a/b/I' — ignore case — GNU; also accepted by current macOS sed
sed -i 's/a/b/g' f — edit in place — GNU only; BSD reads the next word as a backup suffix
sed -i '' 's/a/b/g' f — edit in place, no backup — macOS/BSD only
sed -i.bak 's/a/b/g' f — edit in place, keep f.bak — both: the portable form
sed -E 's/(\w+):(\w+)/\2:\1/' — extended regex with bare ( ); \1 is group 1, & the whole match — -E in both; \w is GNU only
sed 's/a\+/b/' — \+ in a basic regex — GNU only; write sed -E 's/a+/b/'
sed 's\ — /usr\ — /opt\ — g' — any delimiter; no escaping of / — both
sed -n '5p', '5,10p', '5,$p' — print only these lines — both
sed -n '/start/,/end/p' — print between two patterns — both
sed '5d', '/^#/d', '/^$/d' — delete a line, comments, blank lines — both
sed '/^srv/s/80/90/', '10,20s/a/b/' — substitute only on some lines — both
sed '1i\ then the text on the next line — insert before line 1 (a\ appends, c\ changes) — both; one-line 1i text is GNU only
sed 's/[[:space:]]*$//' — strip trailing spaces — both
sed 's/\r$//' — Windows line endings to Unix — GNU, and current macOS sed accepts it too; tr -d '\r' out is the portable form

awk

awk splits each record (line) into fields and runs every pattern { action } on it, in order, then END once. Step through a log: watch the fields, the patterns and the arrays that build the report.

awk '{print $1}', '{print $NF}' — first field; last field
awk -F, '{print $2}', -F'\t', -F'"' — comma, tab or double-quote separated
awk 'NR > 1' — skip the header line
awk '$3 > 100' — lines whose third field is over 100
awk '/error/ {print $2}' — field 2 of matching lines
awk '{s += $1} END {print s}' — sum a column
awk '{s += $1} END {print s / NR}' — average
awk '{c[$1]++} END {for (k in c) print k, c[k]}' — count per key (a group-by); pipe to sort, the order is unspecified
awk '!seen[$0]++' — remove duplicate lines, keep the order
awk 'length($0) > 80' — long lines
awk 'BEGIN {OFS=","} {$1=$1; print}' — rewrite with a new output separator
awk -v lim="$n" '$2 > lim' — pass a shell value in; never splice it into the program text
awk '{print toupper($1)}' — built-ins: length substr index split sub gsub match toupper tolower sprintf

$0, $1, $NF — the line, field 1, the last field
NF — number of fields on this line
NR, FNR — line number overall; within the current file
FS, OFS — input and output field separators
RS, ORS — input and output record separators
FILENAME — the current input file

> [!TIP] Debian and Ubuntu install mawk as awk, not gawk. It has the POSIX functions but not gawk's extensions: gensub() and asort() fail with "function never defined", and PROCINFO is just an empty array. A field that does not look like a number is compared as a string, so $4 >= 500 is true for abc.

sort, uniq, cut, tr and friends

sort -n, -r, -u — numeric, reverse, unique — both
sort -h — human sizes: 2K 10K 1M 3G — both
sort -V — versions: v1.9 before v1.10 — both on current macOS
sort -t, -k3,3n — by field 3 only, numeric, comma separated — both
sort -k1,1nr -k2,2 — by count descending, ties by name: a stable top-N — both
LC_ALL=C sort — byte order: fast and the same on every machine — both
uniq, uniq -c — merge adjacent duplicates; with counts (sort first) — both
uniq -d, -u — only repeated lines; only single lines — both
cut -d, -f1,3, -f2- — fields by a one-character delimiter — both
cut -c1-10 — characters 1 to 10 — both
tr a-z A-Z, tr -d '\r', tr -s ' ' — map, delete, squeeze characters — both
paste -sd, - — join lines with commas — both
comm -13 out.log 2>&1 & — keep running after the terminal closes — both
disown %1 — drop a job from the shell's job list — both
timeout 30 cmd — TERM after 30 seconds; exit 124 if it fired — GNU coreutils; not in macOS by default
flock -n /var/lock/x.lock cmd — run only if no other holder of the lock — util-linux (Linux); not in macOS
lsof -p PID, lsof /path — files a process has open; who has a file open — both
free -h, /proc/PID/ — memory; per-process details — Linux only

Networking and remote work

curl -fsSL url — fail on HTTP 400 and above (exit 22), quiet but show errors, follow redirects
curl -o out url, curl -O url — save to a name; keep the remote name
curl -I url, curl -i url — headers only; headers plus body
curl -w '%{http_code}\n' -o /dev/null -s url — print only the status code
curl -X POST -H 'Content-Type: application/json' -d @body.json url — send JSON from a file
curl --retry 3 --max-time 30 url — retry; overall time limit
ssh -L 8080:localhost:80 host — local forward: your port 8080 reaches port 80 on host
ssh -R 9090:localhost:3000 host — remote forward: host's port 9090 reaches your port 3000
ssh -D 1080 host — a SOCKS proxy on your port 1080
ssh -J bastion host — connect through a jump host
ssh-keygen -t ed25519, ssh-copy-id host — make a key; install it on a server
rsync -avz src/ host:dst/ — copy the contents of src (the trailing / matters)
rsync -avz --delete --dry-run src/ dst/ — preview a mirror before deleting anything
dig +short name, dig name MX — DNS lookups
nc -zv host 443 — is the port open?

What is listening? — ss -tlnp (TCP, listening, numeric, process) — lsof -nP -iTCP -sTCP:LISTEN
Who holds port 8080? — ss -tlnp 'sport = :8080' or lsof -i :8080 — lsof -i :8080
Addresses and routes — ip -br a, ip r — ifconfig, netstat -rn

Your terminal, cron, systemd and CI are different shells

Which files Bash reads depends on how it starts. A script reads none of your startup files, so anything your ~/.bashrc sets up is missing under cron, systemd and CI.

Shell — your login shell — /bin/sh unless the crontab sets SHELL= — none: ExecStart= runs the program directly; pipes and > need bash -c or a script — bash -e {0}; with shell: bash, bash --noprofile --norc -eo pipefail {0}
Startup files — read — none — none — none
PATH — yours — a short default — systemd's default, or Environment=PATH=… — the runner image's
Working directory — where you are — the user's home — / for system units, unless WorkingDirectory= — the checkout, $GITHUB_WORKSPACE
Output — the terminal — mailed to MAILTO, or lost — the journal: journalctl -u name — the job log
A terminal to prompt on — yes — no — no — no
Overlapping runs — possible: wrap in flock -n — a running unit is not started again — each run is a separate job

tool — your dotfiles are not read, so your PATH changes are missing — full paths, or set PATH= at the top of the crontab
date +%F — % in a crontab line means newline — escape it as \%, or move the command into a script
Bash syntax — the job runs under /bin/sh (dash on Debian and Ubuntu) — set SHELL=/bin/bash or call a script with a Bash shebang
two runs at once — the job takes longer than its interval — flock -n /var/lock/job.lock script.sh

Debugging and ShellCheck

bash -n script.sh — check syntax without running
bash -x script.sh — print each command after expansion
set -x ... set +x — trace one section
PS4='+ ${BASH_SOURCE}:${LINENO}: ' — put the file and line in the trace
trap 'echo "line $LINENO: $BASH_COMMAND" >&2' ERR — report the failing command
false \ — true; echo "${PIPESTATUS[@]}" — 1 0: status of every stage (read it at once)
declare -p var — the exact value and type
printf '%q\n' "$var" — show hidden spaces and control characters
shellcheck script.sh — find quoting and logic bugs before running

ShellCheck reads the script without running it and explains each finding on its wiki (https://www.shellcheck.net/wiki/SC2086). These are the findings that matter most, with ShellCheck 0.11's own wording.

SC2086 — Double quote to prevent globbing and word splitting. — rm $files — rm -- "$file", or an array
SC2046 — Quote this to prevent word splitting. — kill $(pgrep app) — mapfile -t pids = 500 {print $1}' access.log \ — sort \ — uniq -c \ — sort -rn \ — head
Biggest entries here — du -sh * \ — sort -rh \ — head
Rename *.txt to *.md — for f in *.txt; do mv -- "$f" "${f%.txt}.md"; done
Replace text in many files (GNU) — grep -rl --null old . \ — xargs -0 sed -i 's/old/new/g'
Same on macOS (its grep -Z means decompress) — grep -rl --null old . \ — xargs -0 sed -i '' 's/old/new/g'
Lines in b but not in a — comm -13 <(sort a) <(sort b)
Sum a column — awk '{s += $1} END {print s}' f
Follow a log for errors — tail -F app.log \ — grep --line-buffered ERROR
A random secret — openssl rand -hex 32
Timestamped backup — cp app.yml "app.yml.$(date +%Y%m%d-%H%M%S)"
Strip comments and blank lines — grep -vE '^[[:space:]]*(#\ — $)' app.conf
Run only one copy from cron — flock -n /var/lock/job.lock /usr/local/bin/job.sh

GNU and macOS at a glance

Every macOS entry below was checked with the stock /usr/bin tools on macOS 26.

Edit a file in place — sed -i 's/a/b/' f — sed -i '' 's/a/b/' f
Portable in-place edit — sed -i.bak ... && rm f.bak — the same
\w, \+ in sed — work — not supported; use [[:alnum:]_] and -E with +
Perl regex — grep -P — not available; use -E or perl
File size — stat -c %s f — stat -f %z f
Yesterday's date — date -d yesterday +%F — date -v-1d +%F
Set a file's time — touch -d '3 days ago' f — touch -t 202610020000 f
Custom find output — find -printf — find ... -exec stat -f ... {} +
Empty input to xargs — runs once unless -r — never runs
Sort processes by memory — ps aux --sort=-%mem — ps aux -m
Listening sockets — ss -tlnp — lsof -nP -iTCP -sTCP:LISTEN
Reverse a file — tac f — tail -r f
Time limit on a command — timeout 30 cmd — not built in
Lock a cron job — flock -n lock cmd — not built in
du one level deep — du -d 1 or --max-depth=1 — du -d 1 only
Bash version — usually 5.x — /bin/bash is 3.2, and /bin/sh runs it too

What you will be able to do

Predict exactly which words a command receives, by walking the expansion order.
Draw the fd table for any redirection and pipeline, and say where each stream ends up.
Explain $?, PIPESTATUS, pipefail, the 141 from … | head and every set -e exception.
Keep loop variables alive by choosing between a pipe, < <(…) and lastpipe.
Build grep, sed, awk, sort and find pipelines for log and file questions, and say which flags differ on macOS.
Write cron, systemd and CI jobs that fail loudly, never overlap and clean up after themselves.

When to use it

Writing a deploy, CI or automation script that must stop at the first failure
Answering a log question on the command line: count, rank, sum or extract a column
A script that works in your terminal but fails under cron, CI, systemd or /bin/sh
Bulk file work where names may contain spaces and a wrong glob would delete real data
Finding what holds a port, what uses memory, or which process has a file open
Moving a script between Linux and macOS, where sed, stat, date and ps take different flags
Triage on a Linux host during an incident: count, rank and filter a log in one pipeline
Scheduled jobs (cron, systemd timers, CI) that must lock, retry, clean up and report failures