fpart
Generate partitioned file lists for backups
SYNOPSIS
fpart [options] [path...]
Outputs partition lists to stdout or files.
PARAMETERS
-1, --one-file-per-child
One file per partition line (for find -exec)
-h, --help
Show help message and exit
-l LEN, --line-bytes-limit=LEN
Max bytes per output line
-n NUM, --number-partitions=NUM
Exact number of partitions
-o FILE, --output-file=FILE
Output partitions to FILE (default: stdout)
-s SIZE, --size-partition=SIZE
Max size per partition (e.g., 1G, 500M)
-t TYPE, --type-sort=TYPE
Sort by TYPE: name|size|atime|ctime|mtime (default: size)
-u, --unique
List hard-linked files only once
-v, --verbose
Increase verbosity (repeat for more)
-V, --version
Print version and exit
-x, --no-cross-filesystems
Skip mount points (stay on same filesystem)
DESCRIPTION
fpart is a lightweight filetree partitioner that scans directories and splits large file hierarchies into manageable partitions based on size limits, number of parts, or other criteria. It outputs lists of files suitable for piping into tools like rsync, tar, or find for parallel backups, transfers, or processing.
Designed for efficiency, fpart sorts files in-place to minimize memory usage, handles hard links uniquely with -u, excludes unwanted paths or types, and supports sorting by name, size, or access/modification times. Ideal for distributing workloads across multiple processes or machines, it avoids common pitfalls of naive splitting like uneven partitions or broken hard links.
Usage typically involves scanning a path, generating partition files, then processing each in parallel, e.g., with xargs and nice. It's particularly valuable for petabyte-scale filesystems where sequential tools fail due to timeouts or bandwidth limits.
CAVEATS
Loads entire filetree into memory; huge directories (>millions of files) may exhaust RAM. No built-in filtering—use find or exclusions beforehand. Partitions are approximate when using -s with sorting.
TYPICAL USAGE
fpart -s 1G -o parts.txt /data
xargs -n1 -P8 -I{} nice -n10 rsync -av {} user@remote:/backup/
SIZE UNITS
Supports K,M,G,T (e.g., 500M=512MiB; uses 1024 base)
HISTORY
Developed by Daniel Lacourt starting 2012; open-source on GitHub. Focuses on performance for massive filesystems, with releases improving sorting and memory efficiency up to v1.4.0 (2023).
