rdfind

Find duplicates

$ rdfind [directory]

$ rdfind [dir1] [dir2]

$ rdfind -deleteduplicates true [directory]

$ rdfind -makehardlinks true [directory]

$ rdfind -makesymlinks true [directory]

$ rdfind -dryrun true [directory]

$ rdfind -outputname [results.txt] [directory]

rdfind [-deleteduplicates true] [-makehardlinks true] [options] directories

rdfind (redundant data find) efficiently locates duplicate files across one or more directory trees using a multi-phase detection algorithm. It first groups files by size, then computes partial checksums on the first bytes of same-sized files, and finally performs full checksums only on files that still match, making it fast even on large file sets.
Once duplicates are identified, rdfind can delete them, replace them with hardlinks (saving disk space while keeping the same path), or replace them with symbolic links. A results file lists all duplicates found for manual review, and the -dryrun flag simulates operations without modifying the filesystem. The first file encountered in the argument order is always kept as the original.

-deleteduplicates BOOL

Delete duplicates.

Replace with hardlinks.

Replace with symlinks.

Simulate only.

Ignore empty files.

Remove same inode files.

Results filename.

Minimum file size.

Hardlinks only on same filesystem. Symlinks may break if original moves. Careful with deletion.

rdfind (really delete find) was created for efficient duplicate detection. Its multi-stage algorithm handles large file sets quickly.

fdupes(1), jdupes(1), duff(1)