trace-cmd

Display the status of tracing system

$ trace-cmd stat

$ trace-cmd list -t

$ trace-cmd start -p [timerlat|osnoise|hwlat|blk|mmiotrace|function_graph|wakeup_dl|wakeup_rt|wakeup|function|nop]

$ trace-cmd show

$ trace-cmd stop

$ trace-cmd clear

$ trace-cmd reset

trace-cmd command [options] [arguments]

Examples:
    trace-cmd record -e syscalls -e sched my_program
    trace-cmd report
    trace-cmd list -e

record
    Records trace data based on specified events, functions, or plugins. Often used with a command to execute for tracing duration.

report
    Analyzes and displays recorded trace data from a trace.dat file. Offers various display options and filters.

list
    Lists available ftrace events, plugins, functions, or instances, providing an overview of what can be traced.

start
    Starts kernel tracing without executing a specific command. Trace data is written to the kernel's trace buffer.

stop
    Stops ongoing kernel tracing. Use after trace-cmd start.

extract
    Extracts the contents of the kernel's trace buffer into a trace.dat file. Use after trace-cmd stop.

reset
    Disables all ftrace events and clears the kernel's trace buffer, effectively resetting the tracing system.

options
    Displays various ftrace options and their current status.

check
    Performs checks on the ftrace environment to ensure it's properly configured and available.

-v
    Increases verbosity of output. Can often be specified multiple times for more detailed information.

-d
    Enable debug mode for the trace-cmd command itself.

-h, --help
    Displays a help message for the general command or a specific subcommand.

trace-cmd is a powerful user-space utility that acts as a front-end for the Linux kernel's ftrace tracing framework.

It significantly simplifies the process of collecting, analyzing, and reporting kernel trace data, which can be invaluable for debugging performance issues, understanding kernel behavior, and developing new features. While ftrace itself exposes a complex set of files in /sys/kernel/debug/tracing, trace-cmd abstracts away much of this complexity, allowing users to specify events, start and stop tracing, and generate human-readable reports with simple commands.

It supports various tracing capabilities, including function tracing, event tracing (e.g., syscalls, scheduling, disk I/O), and specific plugins. The data is typically recorded into a binary file (defaulting to trace.dat) which can then be parsed and displayed in various formats.

trace-cmd typically requires root privileges to operate, as it interacts directly with kernel tracing facilities.

Tracing can introduce overhead, especially if a large number of events are enabled or if the trace buffer is small, potentially affecting system performance.

The size of the generated trace.dat files can grow very large quickly, requiring significant disk space.

trace-cmd can trace a vast array of kernel events, which are categorized (e.g., syscalls, sched, irq, block, net). Users can enable specific events using the -e option with trace-cmd record (e.g., -e syscalls -e sched:sched_switch). Additionally, it supports various ftrace plugins like function (for tracing all kernel function calls), function_graph (for showing call graphs), and printk.

By default, trace-cmd saves recorded data to a binary file named trace.dat in the current directory. This file is highly optimized for storage and can be compressed. The trace-cmd report command then parses this binary file and presents the data in a human-readable text format. The trace-cmd report command itself has many options to filter, sort, and format the output, including options to display timestamps, process IDs, CPU numbers, and specific event fields.

trace-cmd primarily interacts with the kernel's ring buffer for tracing data. When using trace-cmd start/stop/extract, it manages this buffer directly. For more complex tracing scenarios, ftrace supports tracing instances (separate ring buffers), which trace-cmd can also manage via specific options like -i or -N with some subcommands to specify an instance name or max buffer size.

trace-cmd was developed by Steven Rostedt, a prominent Linux kernel developer and one of the primary maintainers of the ftrace kernel tracing infrastructure. It emerged as a necessary user-space tool to provide a more intuitive and feature-rich interface to ftrace, abstracting away the complexities of directly manipulating the /sys/kernel/debug/tracing filesystem. It's part of the kernel-tools package in most Linux distributions and has been widely adopted by developers and system administrators for kernel-level debugging and analysis.

ftrace (kernel), perf(1), strace(1), ltrace(1), dmesg(1), sysstat(1)