bpftrace

List all available probes

$ sudo bpftrace -l

$ sudo bpftrace -e '[tracepoint:raw_syscalls:sys_enter { @[comm] = count(); ]}'

$ sudo bpftrace [path/to/file]

$ sudo bpftrace -e '[tracepoint:raw_syscalls:sys_enter /pid == 123/ { @[comm] = count(); ]}'

$ sudo bpftrace -d -e '[one_line_program]'

$ bpftrace [[-V|--version]]

bpftrace [-options...] [script.bt] | -e 'program'

-e 'program'
    Execute the one-liner or script directly from command line

-f FILE
    Read tracing program from FILE instead of argument

-l [PATTERN]
    List available probes matching PATTERN (e.g., 'kprobe:do_sys_open')

-p PID
    Trace the process with PID only (uprobes/usdt)

-c 'CMD [ARGS]'
    Run CMD and trace it until exit

-t
    Include timestamps in trace output

-v
    Verbose; print resulting eBPF program

-vv
    Very verbose; print kernel compiler log

-q
    Quiet; suppress 'Tracing...' and Ctrl-C messages

-d
    Debug; print all fields of bpf_prog_info

-s
    Symbolicate; prefer BTF or symbolization

-I DIR
    Add DIR to search paths for header files

-o OUTPUT
    Output to file instead of stdout

SCRIPT
    Path to bpftrace script file

bpftrace is a high-level tracing language for Linux eBPF (extended Berkeley Packet Filter), enabling dynamic instrumentation of kernel and user-space for observability, debugging, and performance analysis.

It provides a simple, DTrace-like syntax to attach probes to kernel functions, tracepoints, user-space functions, syscalls, and hardware events. Scripts are compiled just-in-time to eBPF bytecode, loaded into the kernel, and executed safely without modifying kernel code.

Key strengths include one-liners for quick tracing (e.g., counting syscalls), complex scripts for aggregations, histograms, and time-series data, and portability across modern Linux kernels supporting eBPF. It's safer than kernel modules, with verifier protection against crashes.

Common uses: latency analysis (biolatency.bt), off-CPU profiling, network packet inspection, custom metrics. Supports maps for state (hash, array), actions (@ actions), and output formats (print, printf, clear). Ideal for SREs, developers, and kernel hackers needing production-safe tracing.

Requires Linux kernel 4.9+ with eBPF support, CAP_SYS_ADMIN or CAP_BPF; privileged mode often needed. Large programs may hit verifier limits. BTF (BPF Type Format) vmlinux improves symbolication. Not for real-time systems due to JIT compilation overhead.

kprobe:func (kernel entry), kretprobe:func (return), uprobe:/path:offset (user entry), tracepoint:syscalls:sys_enter_open, usdt:path:provider:probe, profile:hz:99 (sampling), software:tcploss, hardware:cache-misses.

bpftrace -e 'tracepoint:syscalls:sys_enter_execve { printf("execve: pid %d cmd %%s\n", pid, comm); }'