Streams, Redirection & Pipes

The shell is not just a place to type commands - it is a programmable pipeline. Once you understand streams and redirection, you stop typing one command at a time and start composing workflows that would take 50 lines of Python in a single line of bash.

The three standard streams

Every process that runs in Linux is automatically wired to three I/O channels, each identified by a file descriptor number:

These are just files. Everything in Linux is a file - including your terminal. That's the key insight that makes redirection possible.

[keyboard] ──► stdin(0) ──► PROCESS ──► stdout(1) ──► [screen]
                                     └──► stderr(2) ──► [screen]

Redirecting output

Writing stdout to a file

ls -la > output.txt        # Overwrite (create or clobber)
ls -la >> output.txt       # Append (never overwrites)

> is destructive - it silently erases whatever was in output.txt. Use >> when you want to accumulate data, like building a log file across multiple runs.

Redirecting stdin

sort < unsorted.txt        # Feed a file into stdin
mysql -u root -p < dump.sql  # Pipe a SQL dump into mysql

Less common than output redirection, but shows up everywhere in automation.

Redirecting stderr

nmap 10.10.10.0/24 2> errors.txt    # Stderr to file, stdout to terminal
nmap 10.10.10.0/24 2>> errors.txt   # Append stderr to file
nmap 10.10.10.0/24 2>/dev/null      # Discard stderr entirely

Why this matters for recon: Many tools spam warnings or deprecation notices to stderr that clutter your output. Redirecting stderr to /dev/null gives you clean results you can actually parse.

Combining stdout and stderr

nmap -sV 10.10.10.5 > results.txt 2>&1   # Both go to the same file
nmap -sV 10.10.10.5 &> results.txt       # Shorthand for the same thing

2>&1 means "redirect FD 2 to wherever FD 1 currently points." Order matters - write it after the stdout redirect.

/dev/null - the black hole

/dev/null is a special device file that discards everything written to it and returns EOF when read. It is the silence operator of Linux.

command > /dev/null          # Discard stdout
command 2>/dev/null          # Discard stderr
command &>/dev/null          # Discard everything
command < /dev/null          # Provide empty stdin

A common recon pattern: suppress all the "no route to host" noise while capturing only the hosts that respond.

for ip in $(seq 1 254); do
  ping -c1 -W1 192.168.1.$ip &>/dev/null && echo "192.168.1.$ip is up"
done

The pipe |

The pipe connects the stdout of one command directly to the stdin of the next - in memory, without writing a temp file. This is the UNIX philosophy: small tools that do one thing, composed together.

command1 | command2 | command3

Practical recon pipeline

# Find all open ports, extract just the port numbers, sort them
nmap -p- --min-rate 5000 10.10.10.5 | grep ^[0-9] | cut -d/ -f1 | sort -n
 
# Enumerate subdomains and filter only live ones
cat subdomains.txt | while read sub; do
  curl -s -o /dev/null -w "%{http_code} $sub\n" "https://$sub.example.com"
done | grep -v "^000"

tee - split the stream

tee reads stdin and writes to both stdout and a file simultaneously. Perfect for when you want to see output in real time AND save it.

nmap -sV 10.10.10.5 | tee nmap_results.txt
# Output appears on screen AND is saved to file
 
# Append mode
nmap -sV 10.10.10.6 | tee -a nmap_results.txt

Process substitution

A slightly advanced feature worth knowing: <(command) creates a virtual file from command output, letting you pass it to programs that expect a filename.

diff <(cat /etc/passwd) <(cat /etc/passwd.backup)
comm -23 <(sort live_hosts.txt) <(sort known_hosts.txt)

Putting it all together

Here is a realistic workflow: you ran a large nmap scan and want to extract a clean list of hosts with open port 80, deduplicate them, and save to a file - while watching progress:

cat nmap_scan.gnmap \
  | grep "80/open" \
  | grep -oP '\d+\.\d+\.\d+\.\d+' \
  | sort -u \
  | tee live_web_hosts.txt \
  | wc -l

Five commands, zero temp files, one clean output. That is the power of pipes.

Key takeaways

• Processes have three streams: stdin (0), stdout (1), stderr (2).
• > overwrites; >> appends; < feeds stdin from a file.
• 2> redirects stderr; 2>&1 merges stderr into stdout.
• /dev/null silently discards any data written to it.
• | connects stdout of one command to stdin of the next - in memory.
• tee splits a stream so you can watch and save at the same time.
• Mastering pipes lets you turn five-step manual workflows into a single, composable command.