Understanding Pipes in Linux: Chaining Commands Together

A pipe in Linux, written as the vertical bar |, connects the output of one command directly to the input of another command, letting you chain multiple commands into a single processing pipeline. For example, ps aux | grep firefox runs ps aux to list all processes, then feeds that output into grep firefox to filter for lines containing “firefox.” Pipes allow small, single-purpose commands to be combined into powerful data-processing chains without any temporary files.

The Unix Philosophy in a Single Character

If there is one feature that captures the essence of Unix and Linux design philosophy, it is the pipe. The idea is deceptively simple: instead of building large, monolithic programs that try to do everything, build small programs that each do one thing well, and provide a mechanism to connect them together so their combined power exceeds what any single program could achieve alone.

The pipe — represented by the | character — is that mechanism. It takes the standard output of one command and feeds it directly as the standard input of the next command, with no temporary file, no manual copying, no intermediate step. The data flows directly from one process to the next, often without ever touching the disk.

This single character has profound implications for how Linux users work. Instead of memorizing one command with a thousand options to accomplish every possible variation of a task, you learn a set of composable tools — grep for filtering, sort for ordering, uniq for deduplication, wc for counting, awk and sed for transformation — and combine them in whatever sequence solves your specific problem. The combinations are limited only by your imagination and understanding of what each tool does.

This article explains how pipes work technically, walks through building pipelines step by step, covers the most useful commands to combine in pipelines, and explains the important distinction between pipes and file redirection.

How Pipes Work: The Technical Mechanism

Standard Streams: The Foundation

Every command-line program in Linux has three standard I/O streams:

• stdin (standard input, file descriptor 0) — where the program reads input from (by default, your keyboard)
• stdout (standard output, file descriptor 1) — where the program writes normal output (by default, your terminal screen)
• stderr (standard error, file descriptor 2) — where the program writes error messages (by default, also your terminal screen)

A pipe connects the stdout of one process directly to the stdin of the next process. The kernel creates an actual pipe (a unidirectional data channel) connecting the two processes’ file descriptors:

Process A          Pipe (kernel buffer)          Process B
  stdout    ─────────────────────────>   stdin

The Kernel’s Role

When you run command1 | command2, the shell:

1. Creates a pipe (an in-kernel memory buffer with a read end and a write end)
2. Starts command1 with its stdout connected to the pipe’s write end
3. Starts command2 with its stdin connected to the pipe’s read end
4. Both processes run simultaneously — command2 can start processing data as soon as command1 produces it, without waiting for command1 to finish completely

This simultaneous execution is important: for large data streams, the receiving command can begin working immediately rather than waiting for the entire output to be generated first. The pipe buffer (typically 64KB on Linux) provides some slack — if the writer produces faster than the reader consumes, data queues in the buffer; if the buffer fills, the writer blocks until the reader catches up.

A Simple Example

$ ls -la | grep ".txt"
-rw-r--r-- 1 sarah sarah  2048 Feb 18 10:15 notes.txt
-rw-r--r-- 1 sarea sarah  8192 Feb 17 14:30 report.txt

Here, ls -la lists all files with details. Its output is piped to grep ".txt", which filters for lines containing “.txt”. Without the pipe, you would need to run ls -la, manually scan the output, and pick out lines with “.txt” yourself.

Building Pipelines Step by Step

The best way to understand pipes is to build a pipeline incrementally, checking the output at each stage.

Step 1: Start with One Command

$ ps aux
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0 168752 13288 ?        Ss   Feb13   0:05 /sbin/init
sarah     1523  2.1  1.2 2845672 201344 ?      Sl   09:22   0:45 firefox
sarah     2145  0.0  0.1  23456   8912 pts/1   Ss   09:30   0:00 bash

ps aux lists all processes — potentially hundreds of lines.

Step 2: Add a Filter

$ ps aux | grep firefox
sarah     1523  2.1  1.2 2845672 201344 ?      Sl   09:22   0:45 firefox
sarah     1598  3.5  0.8 1234568 135280 pts/0  Sl   09:25   1:23 firefox --renderer

Adding | grep firefox filters to only lines containing “firefox.”

Step 3: Add Another Stage

$ ps aux | grep firefox | wc -l
2

Adding | wc -l counts the number of matching lines — answering “how many firefox processes are running?” in one command.

Step 4: Refine Further

$ ps aux | grep firefox | grep -v grep | wc -l
2

Adding | grep -v grep removes the grep firefox command itself from the results (since grep firefox search process also contains the word “firefox” and would otherwise match its own search, appearing in results before it exits).

Each stage of the pipeline is a simple, understandable transformation. The combination answers a specific, useful question that no single command directly provides.

Essential Commands for Pipelines

Certain commands appear constantly in pipelines because they perform fundamental, composable operations on text data.

grep: Filtering Lines

$ cat access.log | grep "404"                    # Lines containing "404"
$ cat access.log | grep -v "404"                  # Lines NOT containing "404"
$ cat access.log | grep -i "error"                # Case-insensitive search
$ cat access.log | grep -c "404"                   # Count matching lines
$ cat access.log | grep -E "40[0-9]"              # Extended regex: 400-409

sort: Ordering Lines

$ cat names.txt | sort                             # Alphabetical
$ cat numbers.txt | sort -n                        # Numerical order
$ cat data.txt | sort -r                           # Reverse order
$ du -sh */ | sort -rh                             # Sort human-readable sizes, largest first
$ cat file.txt | sort -k2                          # Sort by second field

uniq: Removing Duplicates

$ cat names.txt | sort | uniq                      # Remove duplicate lines (must be sorted first)
$ cat names.txt | sort | uniq -c                   # Count occurrences of each unique line
$ cat names.txt | sort | uniq -d                   # Show ONLY duplicated lines

uniq only removes adjacent duplicate lines, which is why sort typically precedes it — sorting brings duplicates together.

wc: Counting

$ cat file.txt | wc -l                             # Count lines
$ cat file.txt | wc -w                             # Count words
$ cat file.txt | wc -c                             # Count bytes/characters
$ ls | wc -l                                       # Count files in directory

head and tail: Limiting Output

$ cat bigfile.txt | head -20                       # First 20 lines
$ cat bigfile.txt | tail -20                       # Last 20 lines
$ ps aux --sort=-%cpu | head -6                    # Top 5 CPU-using processes (plus header)

cut: Extracting Columns

$ cat /etc/passwd | cut -d: -f1                    # First field (username), delimited by :
$ cat /etc/passwd | cut -d: -f1,7                  # Username and shell fields
$ echo "a,b,c,d" | cut -d, -f2-3                   # Fields 2 through 3: b,c

awk: Powerful Field Processing

$ ps aux | awk '{print $2, $11}'                   # Print columns 2 (PID) and 11 (command)
$ cat data.csv | awk -F, '{sum += $3} END {print sum}'  # Sum third CSV column
$ df -h | awk '$5+0 > 80 {print $6}'               # Show filesystems over 80% used

sed: Stream Editing

$ cat file.txt | sed 's/old/new/'                  # Replace first occurrence per line
$ cat file.txt | sed 's/old/new/g'                 # Replace all occurrences
$ cat file.txt | sed '/^#/d'                       # Delete comment lines
$ cat file.txt | sed -n '5,10p'                    # Print only lines 5-10

tr: Character Translation

$ echo "HELLO" | tr 'A-Z' 'a-z'                    # Convert to lowercase: hello
$ cat file.txt | tr -d '\r'                        # Delete carriage returns (Windows line endings)
$ cat file.txt | tr -s ' '                          # Squeeze multiple spaces into one

xargs: Converting Input Into Arguments

$ find . -name "*.tmp" | xargs rm                  # Delete every found file
$ cat filelist.txt | xargs -I{} cp {} /backup/     # Copy each listed file
$ echo "file1 file2 file3" | xargs -n1 echo         # Process one argument at a time

xargs is unique among these commands — instead of processing piped input as data to transform, it converts piped input into command-line arguments for another command, bridging the gap between commands that produce filenames and commands that expect filenames as arguments rather than stdin.

Practical Pipeline Examples

Find the Top 10 Largest Files in a Directory Tree

$ find . -type f -exec du -h {} \; | sort -rh | head -10

Count Unique IP Addresses in a Web Server Log

$ cat access.log | awk '{print $1}' | sort | uniq -c | sort -rn | head -10

Breaking this down: extract the first field (IP address) from each line, sort them, count unique occurrences, sort by count descending, show the top 10.

Find Which Users Are Currently Logged In and Count Sessions Per User

$ who | awk '{print $1}' | sort | uniq -c | sort -rn

Show Disk Usage of Subdirectories, Sorted, Excluding Hidden Directories

$ du -h --max-depth=1 | grep -v '/\.' | sort -rh

Find All Processes Using More Than 5% Memory

$ ps aux | awk '$4 > 5.0 {print $2, $4, $11}'

Search for a Pattern Across Multiple Files and Show Only Filenames

$ grep -rl "TODO" ~/projects/ | sort

Count Lines of Code Across All Python Files

$ find . -name "*.py" | xargs wc -l | tail -1

Show the 5 Most Recently Modified Files

$ ls -lt | head -6

Extract Email Addresses from a Text File

$ cat document.txt | grep -oE '[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}'

Monitor a Log File for Errors in Real Time

$ tail -f /var/log/syslog | grep --line-buffered "ERROR"

--line-buffered ensures grep outputs matches immediately rather than waiting to buffer a full block — essential when piping from a continuous stream like tail -f.

Chaining Many Commands Together

Pipelines can chain as many commands as needed — there is no practical limit:

$ cat access.log \
    | grep "GET" \
    | awk '{print $7}' \
    | sort \
    | uniq -c \
    | sort -rn \
    | head -20

This finds the 20 most frequently requested URLs in a web server log:

1. grep "GET" — filter to GET requests only
2. awk '{print $7}' — extract the requested path (7th field in common log format)
3. sort — sort alphabetically (needed before uniq)
4. uniq -c — count occurrences of each unique path
5. sort -rn — sort numerically, descending, by count
6. head -20 — show only the top 20

The backslash-newline continuation (\ at line end) lets you format long pipelines across multiple lines for readability — the shell treats it as one continuous command.

Pipes vs. Redirection: An Important Distinction

Pipes and redirection are related but different concepts that are often confused.

Redirection: Connecting to Files

Redirection (>, >>, <) connects a command’s input or output to a file:

$ ls -la > filelist.txt          # stdout goes to a file
$ sort < unsorted.txt            # stdin comes from a file
$ command 2> errors.log          # stderr goes to a file

Pipes: Connecting Commands to Commands

Pipes (|) connect one command’s stdout directly to another command’s stdin — no file involved:

$ ls -la | sort                   # stdout of ls becomes stdin of sort

Combining Both

Pipes and redirection frequently combine in the same command line:

$ cat access.log | grep "ERROR" | sort > errors_sorted.txt

Here, two commands are piped together, and the final result is redirected to a file rather than displayed on screen.

$ command1 2>&1 | command2

This redirects command1‘s stderr to the same place as its stdout (2>&1), and then pipes that combined stream to command2 — ensuring command2 sees both normal output and error messages.

Pipe-Related Special Cases

Piping stderr as Well as stdout

By default, a pipe only carries stdout — stderr still goes to the terminal:

$ command_with_errors | grep "pattern"

If command_with_errors writes to stderr, those messages appear on your screen but bypass the grep filter entirely. To include stderr in the pipe:

$ command_with_errors 2>&1 | grep "pattern"

The Exit Status of a Pipeline

By default, $? after a pipeline reflects only the last command’s exit status, not whether earlier commands in the pipe succeeded:

$ false | true
$ echo $?
0    # Reports success, even though 'false' failed!

To check whether any command in the pipeline failed, use PIPESTATUS (Bash-specific):

$ false | true
$ echo ${PIPESTATUS[@]}
1 0    # Shows exit status of EACH command in the pipe

Or use set -o pipefail in scripts to make the pipeline’s overall exit status reflect the first failing command:

#!/bin/bash
set -o pipefail
false | true
echo $?    # Now reports 1 (failure), because pipefail is set

This is important in scripts where you need to detect failures anywhere in a pipeline, not just in the final command.

Named Pipes (FIFOs): Persistent Pipes

The | operator creates an anonymous, temporary pipe that exists only for the duration of one command line. Linux also supports named pipes (FIFOs) — pipe-like objects that exist as actual filesystem entries and can be used across separate command invocations:

$ mkfifo mypipe
$ echo "hello" > mypipe &     # Write to the pipe (backgrounded, blocks until read)
$ cat mypipe                  # Read from the pipe
hello

Named pipes are used for inter-process communication between unrelated processes or scripts that need to exchange data without a regular anonymous pipe’s single-command-line scope.

Process Substitution: Pipe-Like Behavior with Multiple Inputs

Process substitution <(...) lets you use a command’s output as if it were a file — useful when a command needs a filename argument rather than stdin:

$ diff <(ls dir1) <(ls dir2)          # Compare directory listings without temp files
$ comm <(sort file1.txt) <(sort file2.txt)   # Compare two sorted streams

This is conceptually related to pipes (connecting command output to another command) but works around the limitation that some commands need actual file arguments rather than accepting stdin.

Common Pipeline Mistakes

Forgetting That grep Needs sort Before uniq

$ cat data.txt | uniq -c              # WRONG: only removes adjacent duplicates
$ cat data.txt | sort | uniq -c       # CORRECT: sort brings duplicates together first

Piping Output That Is Not Actually Line-Based Text

Pipes work with any data stream, but text-processing tools like grep, sort, and awk expect line-based text. Piping binary data through them produces unpredictable results:

$ cat image.jpg | grep "something"    # Meaningless — binary data, not text lines

Overusing cat (Useless Use of cat)

A very common pattern that experienced users avoid:

$ cat file.txt | grep "pattern"       # Works, but unnecessary
$ grep "pattern" file.txt             # Better: grep can read files directly

Most text-processing commands (grep, sort, awk, sed, wc) can take a filename directly as an argument, eliminating the need for cat to feed them via a pipe. This is purely a style/efficiency consideration — both work correctly, but the direct form avoids spawning an unnecessary extra process.

When cat in a pipeline IS appropriate: combining multiple files (cat file1.txt file2.txt | sort), or when the pipeline’s readability benefits from an explicit starting point.

Quick Reference: Common Pipeline Building Blocks

Conclusion: Small Tools, Unlimited Combinations

Pipes embody the most powerful idea in Unix design: rather than building one tool that does everything, build many tools that each do one thing precisely, and give users a simple, universal way to combine them. The vertical bar character connects any command’s output to any other command’s input, and from that simple mechanism emerges an almost unlimited capacity for data processing — filtering, sorting, counting, transforming, and analyzing text in whatever sequence solves your specific problem.

Learning to build pipelines is less about memorizing specific combinations and more about understanding what each tool contributes: grep filters, sort orders, uniq deduplicates, wc counts, awk and sed transform, xargs bridges the gap to commands expecting arguments rather than stdin. Once these building blocks are second nature, you can construct a pipeline for almost any text-processing task by reasoning through the steps: what do I have, what do I want, and what sequence of transformations gets me there.

This compositional power, available in every Linux terminal with no special software required, is one of the most practically valuable skills in the entire Linux toolkit.