Text Processing Tools

grep, sed, awk, cut, sort, uniq, tr, wc, head/tail, and xargs for command-line data processing.

Overview

Overview The Unix text processing toolkit is a core skill for Linux administration and DevOps. These tools are composable via pipes and form the basis of shell-based data processing pipelines.

grep family: grep searches for patterns. -E (or egrep) enables extended regex (|, +, ?, {n,m}, ()). -F (or fgrep) does fixed-string matching (faster, no regex). Key flags: -v (invert, print non-matching), -r (recursive directory search), -l (print filenames only), -n (line numbers), -c (count matches), -i (case-insensitive), -o (print only matching part), -A/-B/-C N (after/before/context lines).

sed (stream editor): Processes text line by line. The s command is most common: sed 's/pattern/replacement/g'. Flags on s: g (global, all occurrences on line), i (case insensitive), p (print, use with -n). -i for in-place editing (use -i.bak on macOS for backup). Address ranges: sed '3,10d' (delete lines 3-10), sed '/start/,/end/d'.

awk: A full programming language for field-based text processing. Field separator: -F ':' or FS=":" inside. Fields: $1 $2 ... $NF (last field). NR=line number, NF=number of fields. BEGIN{} runs before input, END{} runs after. printf for formatted output. Ideal for summing columns, filtering on conditions, and reformatting structured text.

Supporting tools: cut -d: -f1,3 (extract specific fields by delimiter), sort -k2 -n (numeric sort on field 2), sort -u (unique), uniq -c (count duplicates — requires sorted input), tr -d '\r' (delete carriage returns), tr -s ' ' (squeeze repeated spaces), wc -l (line count), head -n 20, tail -n 20, tail -f (follow), xargs -I{} command {} (pipe list to command arguments).

Key concepts

grep -E extended regex
grep -E "pattern1|pattern2" — enables |, +, ?, (), {n,m}. Equivalent to egrep. grep -F for literal string (no regex), much faster for fixed patterns.
sed s/pattern/replace/
sed s/old/new/g — global replace. -i for in-place. Address ranges: /regex/s/old/new/. sed -n p with -n suppresses default print. d deletes lines. p prints matched lines.
awk field processing
awk -F: '{print $1, $3}' — print fields. BEGIN{}/END{} for setup/teardown. $NF=last field. sum+=$1 in body, print sum in END for column sums. printf for formatted output.
xargs for parallelism
xargs -I{} command {} for substitution. xargs -P4 for 4 parallel processes. xargs -n1 for one arg per invocation. find ... | xargs grep is much faster than grep -r for large trees.

When to use it

  • Analyzing log files for errors and patterns
  • Extracting specific fields from structured text (CSV, colon-delimited)
  • Finding and replacing text across many files
  • Building reporting pipelines from command output

Interview questions

  • You have a log file where each line is an Apache access log entry. Write a one-liner to find the top 10 IP addresses by request count.
  • Explain the difference between grep, egrep, and fgrep. When would you use each?

Sign up free to read the full topic

More Linux topics

Start free on Cariara · Coding problems · System design studio