strace

Start tracing a specific process by its PID

$ strace [[-p|--attach]] [pid]

$ strace [[-p|--attach]] [pid] -e [system_call,system_call2,...]

$ strace [[-p|--attach]] [pid] [[-c|--summary-only]]

$ strace [[-p|--attach]] [pid] [[-T|--syscall-times]] [[-s|--string-limit]] [32]

$ strace [program]

$ strace -e trace=file [program]

$ strace [[-f|--follow-forks]] -e trace=network [[-o|--output]] [trace.txt] [program]

strace [-options] command [arguments...]
strace [-options] -p pid

-c
    Count time, calls, and errors for each system call and report a summary on program exit.

-f
    Trace child processes as they are forked by the traced program.

-o filename
    Write the trace output to filename instead of standard error.

-p pid
    Attach to the process with the ID pid and trace its activity. This requires appropriate permissions.

-e expr
    Specify a qualifying expression to control which events to trace. For example, trace=open,close to trace only open and close system calls, or file=myfile.txt to trace operations on a specific file.

-v
    Enable verbose mode for decoding structures, showing more detailed information for system call arguments.

-tt
    Print a time stamp with microseconds for each line of output, indicating when the system call occurred.

-T
    Show the time spent in each system call, from its entry to its exit. This helps identify performance bottlenecks.

-s strsize
    Specify the maximum string size to print. Longer strings are truncated. Default is 32 characters.

strace is a powerful diagnostic, debugging, and instructional user-space utility for Linux. It intercepts and records the system calls invoked by a program and the signals received by it. This allows users to observe the interactions between a program and the Linux kernel, providing insights into file operations, network communications, process management, memory allocation, and more.

It's invaluable for troubleshooting problems where a program isn't behaving as expected, helping identify missing files, permission issues, or unexpected network activity. strace works by utilizing the ptrace system call, which allows one process to control another, inspect and change its core image, and set breakpoints, thus providing a low-level view of program execution.

strace can significantly slow down the traced program due to the overhead of intercepting every system call.
Tracing very busy or long-running processes can generate extremely large output files, potentially consuming significant disk space.
Requires appropriate permissions (e.g., CAP_SYS_PTRACE or root privileges) to attach to processes or trace privileged operations.
The output can be very verbose and may require careful filtering or parsing to extract useful information.

Debugging programs that crash or behave unexpectedly.
Understanding why a program fails to start (e.g., missing libraries, incorrect permissions, configuration issues).
Analyzing network communication issues or unexpected file accesses.
Reverse engineering program behavior without source code.
Security auditing to see what files or network connections a program accesses.

The default output for each system call typically includes:

syscall_name(arguments) = return_value

For example:
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
read(3, "root:x:0:0:root:/root:/bin/bash\n", 4096) = 32

strace was originally developed by Richard S. Wall and Paul F. Haase in the early 1990s for SunOS. It was later ported to Linux and has since become an indispensable tool in the Linux ecosystem. Its design is rooted in the ptrace system call mechanism, providing a low-level, yet user-friendly, interface for observing program-kernel interactions. Over the decades, it has evolved significantly, adapting to new kernel features, system calls, and architectures.

ltrace(1), gdb(1), perf(1), ptrace(2)