uniq

Remove adjacent duplicate lines

$ sort [file] | uniq

$ sort [file] | uniq -d

$ sort [file] | uniq -u

$ sort [file] | uniq -c

$ sort [file] | uniq -i

$ sort [file] | uniq -f [N]

$ sort [file] | uniq -s [N]

uniq [options] [input [output]]

uniq filters adjacent matching lines from input, writing unique lines to output. It only compares consecutive lines, so input typically needs to be sorted first.
Commonly used with sort in a pipeline: sort file | uniq removes all duplicates. Adding -c shows frequency counts, useful for analyzing log files or finding common patterns.
The -d option shows only lines that appear more than once, while -u shows only lines appearing exactly once. These are mutually exclusive perspectives on the data.
Field and character skipping options allow ignoring prefixes (like timestamps or line numbers) when comparing for uniqueness.

-c, --count

Prefix lines with occurrence count

Only print duplicate lines, one per group

Print all duplicate lines

Only print unique lines (occurring once)

Ignore case when comparing

Skip first N fields when comparing

Skip first N characters when comparing

Compare only first N characters

Line delimiter is NUL, not newline

Uniq only removes adjacent duplicates. Without sorting first, non-adjacent duplicates remain. Always use sort | uniq or sort -u for true deduplication.
sort -u is often more efficient than sort | uniq as it removes duplicates during sorting rather than in a separate pass.
The count option (-c) prefixes with spaces and count, which may need processing for further use.

sort(1), comm(1), cut(1), wc(1)