rr

Record an application

$ rr record [path/to/binary --arg1 --arg2]

$ rr replay

rr command [options] [arguments]
Common commands:
rr record [options] program [program_arguments]
rr replay [options] [trace_dir]
rr dump [options] [trace_dir]
rr ps
rr cat trace_file

record
    Records the execution of a program, creating a trace directory.

replay
    Replays a previously recorded trace. Often used with -g to launch GDB.

dump [trace_dir]
    Dumps the contents of a trace directory, showing recorded events and execution information.

ps
    Lists active rr recording or replay processes on the system.

cat trace_file
    Prints the raw contents of a specific trace file from a trace directory.

-o DIR, --output-trace-dir=DIR
    (Record) Specifies the directory where the trace data will be stored. Defaults to a unique directory in ~/.local/share/rr/.

--nested
    (Record) Allows rr to be used inside another rr recording, useful for debugging rr itself or its interactions.

--disable-randomization
    (Record) Disables Address Space Layout Randomization (ASLR) for the recorded program, making addresses consistent across replays.

--cpu-affinity
    (Record) Pins the recorded process and its children to a single CPU core, which can sometimes improve performance or reduce overhead.

--single-process
    (Record) Instructs rr to only record the initial process and not any of its children (forked processes).

-g, --gdb
    (Replay) Launches GDB and automatically attaches it to the replaying process. This is the primary way to debug with rr.

-s, --server
    (Replay) Starts a GDB server on a specified port, allowing a remote GDB client to connect to the replaying process.

-f, --on-exit-failure
    (Replay) Executes a specified command if the replayed process exits with a non-zero status or is terminated by a signal.

-v, --verbose
    Increases verbosity of rr's output, useful for debugging rr's own operations.

-h, --help
    Displays help information for rr commands and options.

The rr command is a powerful, lightweight tool for recording and deterministically replaying program execution on Linux. Developed primarily by Mozilla, rr allows developers to capture the exact execution path of a program, including all non-deterministic events like system calls, signals, and context switches. This capability is invaluable for debugging intermittent or hard-to-reproduce bugs, especially concurrency issues (Heisenbugs).

When a program is recorded, rr logs all necessary inputs and events. Later, this recording can be replayed an arbitrary number of times, always yielding the identical execution. This deterministic replay enables powerful debugging workflows, particularly when integrated with the GDB debugger. Users can step forward and backward through code, inspect memory, and set breakpoints, knowing that the execution environment is perfectly reproducible. rr virtualizes CPU time, ensuring that even time-dependent operations behave deterministically during replay. It significantly reduces the time and effort required to diagnose complex software defects.

rr has certain limitations. It primarily works on x86-64 Linux systems and requires specific kernel features (like perf_event_open and ptrace support). While generally efficient, recording can introduce performance overhead, especially for CPU-intensive or highly I/O-bound applications. The generated trace files can also become quite large, requiring significant disk space. Furthermore, rr cannot record everything; it's designed for user-space applications and does not trace kernel execution. It also doesn't support recording processes that use certain advanced kernel features or hardware interactions directly (e.g., direct GPU programming without standard APIs). Recording multi-threaded applications is fully supported and is one of its strongest features for reproducing concurrency bugs, but it does incur more overhead.

The primary use case for rr is in conjunction with the GNU Debugger (GDB). After recording a program's execution using rr record, users can replay the trace and attach GDB (e.g., rr replay -g). This allows for full debugging capabilities, including reverse execution (stepping backward in time), inspecting variables, and setting breakpoints, all with perfect determinism. This is crucial for understanding the root cause of complex bugs.

When rr records a program, it creates a unique trace directory (usually in ~/.local/share/rr/). This directory contains all the necessary data to replay the execution, including syscall logs, memory maps, and other metadata. This self-contained nature makes it easy to share traces with other developers for collaborative debugging.

rr requires a Linux kernel with perf_event_open and ptrace support, running on an x86-64 architecture. While highly optimized, recording introduces some performance overhead, especially for I/O-heavy applications. The size of trace files can also be substantial for long-running or memory-intensive programs, so sufficient disk space is recommended.

The rr project was initiated at Mozilla by Robert O'Callahan in 2011, driven by the need for a more robust and deterministic way to debug complex browser bugs, particularly those involving concurrency. The initial motivation was to tackle Heisenbugs that were notoriously difficult to reproduce in traditional debugging environments. Over the years, it has gained significant traction and contributions from various developers and organizations, including Google. Its development continues to be active, with ongoing improvements in performance, stability, and feature set. rr has become an essential tool in the arsenal of many developers working on intricate C/C++ applications on Linux, proving its value in reproducing and fixing bugs that would otherwise be nearly impossible to diagnose.

gdb(1), valgrind(1), strace(1), ltrace(1), perf(1)