stap

stap [ options ] SCRIPT_FILE [ script-arguments ... ]
stap [ options ] -e SCRIPT_TEXT [ script-arguments ... ]
stap [ options ] -l PROBE_PATTERN

-g, --guru
    Enables guru mode, allowing scripts to use constructs that might be unsafe or unstable.

-v, --verbose
    Increases the verbosity of messages during compilation and execution.

-p N, --pass=N
    Stops stap after a specified compilation pass N (e.g., 1=parse, 2=elab, 3=sconf, 4=xlate, 5=cparse, 6=ld).

-o FILE, --output=FILE
    Directs the script's output to the specified FILE instead of standard output.

-D VAR=VALUE
    Defines a preprocessor macro VAR with an optional VALUE, similar to C's #define.

-I DIR, --search-dir=DIR
    Adds DIR to the list of directories searched for included SystemTap files (e.g., .stp, .h).

-l PROBE, --list-probes=PROBE
    Lists all available probe points matching the given PROBE pattern (e.g., 'kernel.function("sys_*")').

-e SCRIPT
    Executes the SCRIPT text directly from the command line, useful for short one-liner scripts.

-c CMD, --command=CMD
    Runs the specified CMD and terminates the SystemTap script when CMD exits.

-F, --unbuffered
    Disables buffering of output, causing each line to be flushed immediately.

-s N, --buffer-size=N
    Sets the per-CPU buffer size for collected data in kilobytes (default is 4MB per CPU).

-w, --warn-error
    Treats all warnings as errors, causing compilation to fail if warnings are issued.

--sysroot=DIR
    Specifies an alternative system root directory for locating kernel source and debuginfo.

-r RELEASE, --release=RELEASE
    Specifies the kernel release version for which to compile the script, useful for cross-compilation.

stap is the front-end command for SystemTap, a powerful tracing and probing tool for the Linux kernel and user-space applications. It allows developers and system administrators to gather detailed information about system activities in real-time, without recompiling the kernel or application. SystemTap uses a simple scripting language (Stap Language) to define what data to collect, when to collect it (probes), and what to do with it. This makes it invaluable for debugging performance issues, understanding kernel behavior, and troubleshooting complex system problems.

The stap command compiles these scripts into kernel modules, loads them, runs them, and then unloads them automatically or manually. It provides a safe and non-intrusive way to instrument the running system, with robust safeguards to prevent system crashes or instability. It supports a wide range of probing points, including kernel functions, user-space functions, tracepoints, kprobes, performance counters, and more, offering deep visibility into system internals.

Using stap typically requires root privileges. It also necessitates the presence of matching kernel headers and debuginfo packages for the running kernel. While SystemTap is designed with safety in mind and employs a verifier to check scripts, poorly written or malicious scripts, especially when using 'guru mode' (-g), can potentially lead to system instability or crashes. The performance overhead varies greatly depending on the complexity and frequency of the probes; excessive probing can impact system performance.

SystemTap scripts are written in a specialized, event-driven language. It shares similarities with C but is purpose-built for tracing, offering constructs like probes, handlers, and aggregate functions to collect and process data in real-time.

SystemTap can attach probes to a wide variety of locations, including kernel functions, system calls, user-space functions, tracepoints, Kprobes, Uprobes, performance counters, and timers. This allows for extremely granular observation of system behavior.

A core design principle of SystemTap is safety. Before execution, scripts undergo a rigorous verification process (static analysis and sandboxing) to ensure they cannot crash the kernel, loop infinitely, or leak sensitive data unintentionally. This makes it suitable for production environments.

The SystemTap project, with stap as its primary interface, began development around 2005. It was primarily driven by engineers from Red Hat, IBM, and Intel. The motivation was to create a robust, safe, and dynamic tracing framework for the Linux kernel, similar in concept to Solaris's DTrace, but tailored for Linux. It aimed to provide system administrators and developers with a powerful tool for deep kernel and user-space introspection without requiring kernel recompilation or reboot. Over the years, SystemTap has become an integral part of the Linux debugging and performance analysis ecosystem, seeing continuous development to enhance its features, safety, and performance.

dtrace(1), perf(1), strace(1), ltrace(1), stap-merge(1), stap-report(1)