trace-cmd
Trace kernel function execution
TLDR
Display the status of tracing system
List available tracers
Start tracing with a specific plugin
View the trace output
Stop the tracing but retain the buffers
Clear the trace buffers
Clear the trace buffers and stop tracing
SYNOPSIS
trace-cmd command [options] [arguments]
Examples:
trace-cmd record -e syscalls -e sched my_program
trace-cmd report
trace-cmd list -e
PARAMETERS
record
Records trace data based on specified events, functions, or plugins. Often used with a command to execute for tracing duration.
report
Analyzes and displays recorded trace data from a trace.dat file. Offers various display options and filters.
list
Lists available ftrace events, plugins, functions, or instances, providing an overview of what can be traced.
start
Starts kernel tracing without executing a specific command. Trace data is written to the kernel's trace buffer.
stop
Stops ongoing kernel tracing. Use after trace-cmd start.
extract
Extracts the contents of the kernel's trace buffer into a trace.dat file. Use after trace-cmd stop.
reset
Disables all ftrace events and clears the kernel's trace buffer, effectively resetting the tracing system.
options
Displays various ftrace options and their current status.
check
Performs checks on the ftrace environment to ensure it's properly configured and available.
-v
Increases verbosity of output. Can often be specified multiple times for more detailed information.
-d
Enable debug mode for the trace-cmd command itself.
-h, --help
Displays a help message for the general command or a specific subcommand.
DESCRIPTION
trace-cmd is a powerful user-space utility that acts as a front-end for the Linux kernel's ftrace tracing framework.
It significantly simplifies the process of collecting, analyzing, and reporting kernel trace data, which can be invaluable for debugging performance issues, understanding kernel behavior, and developing new features. While ftrace itself exposes a complex set of files in /sys/kernel/debug/tracing, trace-cmd abstracts away much of this complexity, allowing users to specify events, start and stop tracing, and generate human-readable reports with simple commands.
It supports various tracing capabilities, including function tracing, event tracing (e.g., syscalls, scheduling, disk I/O), and specific plugins. The data is typically recorded into a binary file (defaulting to trace.dat) which can then be parsed and displayed in various formats.
CAVEATS
trace-cmd typically requires root privileges to operate, as it interacts directly with kernel tracing facilities.
Tracing can introduce overhead, especially if a large number of events are enabled or if the trace buffer is small, potentially affecting system performance.
The size of the generated trace.dat files can grow very large quickly, requiring significant disk space.
TRACING EVENTS AND PLUGINS
trace-cmd can trace a vast array of kernel events, which are categorized (e.g., syscalls, sched, irq, block, net). Users can enable specific events using the -e option with trace-cmd record (e.g., -e syscalls -e sched:sched_switch). Additionally, it supports various ftrace plugins like function (for tracing all kernel function calls), function_graph (for showing call graphs), and printk.
OUTPUT FILES AND FORMATS
By default, trace-cmd saves recorded data to a binary file named trace.dat in the current directory. This file is highly optimized for storage and can be compressed. The trace-cmd report command then parses this binary file and presents the data in a human-readable text format. The trace-cmd report command itself has many options to filter, sort, and format the output, including options to display timestamps, process IDs, CPU numbers, and specific event fields.
KERNEL BUFFERS AND INSTANCES
trace-cmd primarily interacts with the kernel's ring buffer for tracing data. When using trace-cmd start/stop/extract, it manages this buffer directly. For more complex tracing scenarios, ftrace supports tracing instances (separate ring buffers), which trace-cmd can also manage via specific options like -i or -N with some subcommands to specify an instance name or max buffer size.
HISTORY
trace-cmd was developed by Steven Rostedt, a prominent Linux kernel developer and one of the primary maintainers of the ftrace kernel tracing infrastructure. It emerged as a necessary user-space tool to provide a more intuitive and feature-rich interface to ftrace, abstracting away the complexities of directly manipulating the /sys/kernel/debug/tracing filesystem. It's part of the kernel-tools package in most Linux distributions and has been widely adopted by developers and system administrators for kernel-level debugging and analysis.