sc_wartsfilter

Filter all data records that had specific destinations and write them to a separate file

$ sc_wartsfilter -i [path/to/input.warts] -o [path/to/output.warts] -a [192.0.2.5] -a [192.0.2.6]

$ sc_wartsfilter -i [path/to/input.warts] -o [path/to/output.warts] -a [2001:db8::/32]

$ sc_wartsfilter -i [path/to/input.warts] -t [ping] | sc_warts2json

sc_wartsfilter [options] [wartsfile]

-A
    Filter by source or destination IP address or prefix.

-D
    Filter by destination IP address or prefix.

-S
    Filter by source IP address or prefix.

-a
    Filter by source or destination ASN.

-d
    Filter by destination ASN.

-s
    Filter by source ASN.

-F
    Filter by specific flow ID.

-t :
    Filter by timestamp range (UNIX epoch seconds or YYYYMMDDHHMMSS format). Start or end can be omitted.

-T
    Filter by record type (e.g., ping, tr, cycle, dns).

-M
    Filter out records with RTT greater than max_rtt (milliseconds).

-m
    Filter out records with RTT less than min_rtt (milliseconds).

-H
    Filter out traceroutes with more than max_hops.

-h
    Filter out traceroutes with less than min_hops.

-e
    Exclude records that match the specified criteria (invert filter logic).

-i
    Read input from file instead of standard input.

-o
    Write output to file instead of standard output.

-v
    Enable verbose output, showing progress and matched records.

-P
    Print record header to stdout.

-X
    Print flow records in hexadecimal format.

-x
    Pretty-print warts records to stdout (useful for debugging and human readability).

-w
    Write matching warts records to stdout (this is the default behavior if not using -o, -X, -x, or -P).

sc_wartsfilter is a command-line utility from the CAIDA Scamper suite designed to filter binary Warts files. Warts files store network measurement data, such as traceroutes and pings. This tool allows users to extract specific subsets of data based on various criteria, including source/destination IP addresses or prefixes, Autonomous Systems (ASNs), flow IDs, timestamp ranges, and record types (e.g., traceroute or ping records). It also supports filtering by round-trip times (RTT) and hop counts. It's essential for processing large datasets and isolating relevant measurements for analysis, reading from standard input or a specified file and writing filtered Warts data to standard output or a designated file.

Warts files are a custom binary format, making direct inspection difficult without specific tools like sc_wartsfilter or sc_wartsdump. ASN filtering relies on available IP-to-ASN mappings, which may not be perfectly up-to-date or comprehensive within the context of the Warts file's collection time.

sc_wartsfilter primarily operates on Warts files, a custom binary format developed by CAIDA for storing network measurement data efficiently. The filtered output, by default, is also in the Warts format, preserving the structure of the records. When options like -X or -x are used, the output becomes human-readable representations for debugging or specific analysis.

Multiple filtering options specified on the command line are typically combined with a logical AND operation. For instance, using -S and -D simultaneously will only match records that satisfy both the source and destination criteria. The -e option inverts the final result of this combined logic, effectively excluding records that match all other specified criteria.

sc_wartsfilter is a core component of the Scamper measurement tool suite, developed by CAIDA (Center for Applied Internet Data Analysis). It was created to facilitate the large-scale analysis of active network topology measurements by providing a robust way to process and filter the binary Warts datasets generated by Scamper. Its development is ongoing as part of CAIDA's continuous effort to provide tools for Internet research and network analysis.

scamper(1), sc_wartsdump(1), sc_warts2json(1), sc_warts2csv(1), mtr(8), traceroute(8)