pueue-parallel
Run commands in parallel using a queue
TLDR
Set the maximum number of tasks allowed to run in parallel, in the default group
Set the maximum number of tasks allowed to run in parallel, in a specific group
SYNOPSIS
pueue-parallel [OPTIONS] [COMMAND_TEMPLATE]
(Reads commands from stdin, or uses COMMAND_TEMPLATE with input lines)
PARAMETERS
-j, --jobs
Specify the number of commands to execute in parallel for this batch. This overrides Pueue's global concurrency limit for tasks added by this invocation.
-g, --group
Assign the submitted tasks to a specific Pueue group. This helps in organizing and managing related tasks.
--command-template
Define a command template that will be executed for each input line. Use '{}' as a placeholder for the input line. For example, '--command-template "echo Processing {}"'.
--delay-for
Delay the execution of the submitted tasks by a specified duration (e.g., '5s', '2m', '1h').
--dry-run
Perform a dry run: print the commands that would be submitted to Pueue without actually adding them.
--no-pueue
Execute commands sequentially without using the Pueue daemon. This bypasses Pueue's queueing and parallelization features, acting as a simple sequential runner.
--max-tasks
Limit the maximum number of tasks to be submitted from the input.
--max-args
When using '--command-template', specify the maximum number of input lines to be passed as arguments to a single command.
--progress-bar
Display a progress bar indicating the submission and execution status of the tasks.
--start-immediately
Instruct Pueue to start the submitted tasks immediately, overriding any pending state.
DESCRIPTION
pueue-parallel is a utility that streamlines the execution of multiple commands in parallel using the Pueue task queue system. It is designed to read commands from standard input, typically one command per line, and submit them as individual tasks to the Pueue daemon. Alternatively, it can process input lines by substituting them into a specified command template.
This command is particularly useful for orchestrating large batches of jobs, ensuring that the number of concurrently running tasks adheres to Pueue's configured parallelization limits. It acts as a bridge between common Unix pipeline tools (like find, grep, or cat) and Pueue, allowing users to easily parallelize tasks that might otherwise be run sequentially or managed with more complex scripting. It offers features like specifying job groups, limiting concurrency, and handling input processing, making it a flexible tool for batch processing within the Pueue ecosystem.
CAVEATS
pueue-parallel requires a running Pueue daemon to submit and manage tasks (unless '--no-pueue' is used). It's designed for batch processing where each input line corresponds to a task or part of a task. While powerful, complex command logic or inter-task dependencies are best handled by Pueue's core scheduling features rather than solely relying on this helper. Error handling for individual commands is managed by Pueue's logging and status tracking.
INPUT MODES
pueue-parallel primarily accepts input via standard input (stdin). Each line read from stdin is treated as a command to be executed, or as an argument to be substituted into a COMMAND_TEMPLATE. This allows seamless integration with other commands that output lists, such as find with -print0 (when combined with xargs -0 and a template) or cat. When no COMMAND_TEMPLATE is provided, each input line is run directly as a command.
INTEGRATION EXAMPLES
This command is frequently used in conjunction with other shell utilities to automate parallel processing. For instance, to convert all .jpg files in a directory to .png in parallel:find . -name "*.jpg" | pueue-parallel -j 4 --command-template "convert {} {}.png"
Or to simply run a list of commands defined in a file:cat commands.txt | pueue-parallel
These patterns highlight its utility in building powerful, parallelized workflows.
HISTORY
pueue-parallel is an integral part of the Pueue project, a command-line task queue and scheduler. It was developed to simplify the process of submitting multiple independent tasks to Pueue for parallel execution, particularly when integrating with standard Unix pipelines. Its development reflects the need for a user-friendly wrapper to leverage Pueue's robust scheduling capabilities for batch jobs, making it easier to manage concurrent operations without manual task submission.
