sattach

Redirect the IO streams (stdout, stderr, and stdin) of a Slurm job step to the current terminal

$ sattach [jobid].[stepid]

$ sattach --input-filter [task_number]

$ sattach --[output|error]-filter [task_number]

sattach [OPTIONS] <job_id>[.<step_id>]

-j <jobid>[.<stepid>], --jobid=<jobid>[.<stepid>]
    Specify the Slurm job ID and optionally the job step ID to which to attach. This is the primary argument for identifying the target.

-l, --label
    Prepend all lines of output from the attached job or step with its job ID and step ID for easier identification.

-s, --slim
    Suppress initial informational headers printed by sattach, outputting only the attached job's data.

-x, --exit-on-eof
    Exit sattach when an End-of-File (EOF) is received on any of the standard I/O streams (stdin, stdout, stderr) from the attached job.

-q, --quiet
    Do not print informational messages; only display the job's output.

-p, --pty
    Request a pseudo-terminal (PTY) allocation for the attached job. This is essential for proper interaction with shell-based or interactive applications.

-d, --detach
    Do not send a SIGHUP signal to the attached job or step upon sattach exit. This ensures the job continues running unaffected when sattach terminates.

--clear-input
    Clear the existing input buffer for the job step before attaching to its standard input, preventing stale input from being processed.

--timeout=<seconds>
    Specify the maximum time in seconds to wait for the job step to be ready for attachment. If the timeout is exceeded, sattach will fail.

--verbose
    Increase the verbosity of sattach's output, showing more detailed diagnostic or informational messages.

--version
    Display the sattach version number and exit.

--help
    Display a brief help message summarizing command usage and options, then exit.

sattach is a utility within the Slurm Workload Manager suite designed to connect to the standard input, output, and error streams of an already running job step on a compute node.

This command allows users to observe a job's real-time output, or provide input if the job is interactive, without directly logging into the execution node. It's particularly useful for debugging long-running processes, monitoring progress, or re-establishing a connection to interactive shell sessions launched via srun that may have become detached. sattach requires the job ID and optionally the job step ID to establish the connection, providing a non-intrusive way to interact with distributed jobs.

sattach is part of the Slurm Workload Manager and requires a functional Slurm environment. Users can typically only attach to job steps they own due to security considerations. Attempting to attach to a job's standard input when it's not designed for interactive input or expects input from a file can lead to unexpected behavior or interfere with the job's execution.

sattach is primarily utilized in High-Performance Computing (HPC) environments where Slurm is deployed as the resource manager. It is invaluable for system administrators and researchers who need to interact with or monitor the progress of specific job steps or interactive sessions launched via srun without direct node login.

By default, a user is restricted to attaching only to their own job steps. However, system administrators can configure Slurm (e.g., via the AllowAttach parameter in slurm.conf) to permit other users or groups, such as support staff, to attach to jobs for collaborative debugging or operational support purposes.

sattach is an integral component of the Slurm Workload Manager, which originated in the early 2000s (around 2002-2003) as a robust, open-source job scheduler for HPC clusters. Its functionality has evolved alongside Slurm, providing a critical tool for real-time monitoring, debugging, and interaction with jobs running in distributed environments, addressing the complex needs of modern cluster management.

srun(1), sbatch(1), scancel(1), squeue(1)