cargo-bench

Execute all benchmarks of a package

$ cargo bench

$ cargo bench --no-fail-fast

$ cargo bench --no-run

$ cargo bench --bench [benchmark]

$ cargo bench --profile [profile]

$ cargo bench --examples

$ cargo bench --bins

$ cargo bench --lib

cargo bench [OPTIONS] [BENCH] [-- ARGS...]

BENCH
    If specified, only run the benchmark target with the given name. If omitted, all benchmark targets are run.

--release
    Build artifacts in release mode, with optimizations. This is crucial for meaningful benchmark results.

-p , --package
    Bench only the specified package, useful in a workspace.

--lib
    Bench the package's library.

--bin
    Bench the specified binary target.

--bins
    Bench all binary targets.

--example
    Bench the specified example target.

--examples
    Bench all example targets.

--bench
    Bench the specified benchmark target.

--benches
    Bench all benchmark targets.

--all-targets
    Bench all targets (library, binaries, examples, tests, and benchmarks).

--features
    Space or comma separated list of features to activate.

--all-features
    Activate all available features of all packages in the workspace.

--no-default-features
    Do not activate the `default` feature.

--target
    Build for the target triple (e.g., `x86_64-unknown-linux-gnu`).

-j , --jobs
    Number of parallel jobs to run. Defaults to the number of CPUs.

-v, --verbose
    Use verbose output from Cargo.

-q, --quiet
    Do not print Cargo log messages.

--workspace
    Bench all packages in the workspace.

--profile
    Build with the given profile (e.g., `dev`, `release`, or a custom profile).

-- ...
    Arguments following `--` are passed directly to the benchmark executable. These are often used to configure the benchmarking harness, e.g., `-- --noplot` for Criterion.rs.

The cargo bench command is a fundamental part of the Rust ecosystem's build tool, Cargo. It compiles and executes the benchmarks defined within a Rust project. Unlike cargo test which runs correctness checks, cargo bench focuses on measuring the performance characteristics of code sections, such as execution time, throughput, or memory usage.

Benchmarks in Rust are typically defined in separate `[[bench]]` sections in the `Cargo.toml` file or placed in the `benches/` directory. They rely on external benchmarking harnesses like Criterion.rs or the older bencher crate, as Rust's standard library does not provide a built-in benchmarking framework. When `cargo bench` is invoked, it compiles these benchmark targets and then runs them, often passing control to the chosen benchmarking harness to perform statistical analysis and report results. For accurate and meaningful performance metrics, it is crucial to run benchmarks in release mode using the `--release` flag, as this enables compiler optimizations that are disabled in debug builds.

Benchmarking in Rust, especially with `cargo bench`, requires careful consideration:

1. Harness Requirement: Rust's standard library does not provide a built-in benchmarking harness. You must add an external crate like Criterion.rs (recommended) or bencher as a `dev-dependency` in your `Cargo.toml` and implement your benchmarks using its API.
2. Release Mode: Always run benchmarks with the `--release` flag. Running in debug mode (default) will yield artificially slow results due to lack of compiler optimizations.
3. Environmental Factors: Benchmark results can be highly sensitive to the operating system's state, background processes, CPU frequency scaling, and even temperature. For consistent results, run benchmarks on a stable, isolated system with minimal background activity.
4. Microbenchmarking Pitfalls: Be aware of compiler optimizations that might optimize away code being benchmarked if it has no observable side effects. Ensure your benchmarks are measuring what you intend.

To use `cargo bench`, you first need to configure a benchmarking harness. For Criterion.rs, you typically:

1. Add `criterion` as a `dev-dependency` in your `Cargo.toml` file, for example:
`[dev-dependencies]`
`criterion = { version = "0.5", features = ["html_reports"] }`
Then, configure the benchmark target (if using Criterion as a separate executable, which is common for full features):
`[[bench]]`
`name = "my_benchmarks"`
`harness = false`

2. Create a benchmark file, typically in `benches/my_benchmarks.rs`, and write your benchmark functions using the Criterion API. A simple example:
`use criterion::{criterion_group, criterion_main, Criterion};`
`fn bench_my_function(c: &mut Criterion) {`
`c.bench_function("my_function", |b| b.iter(|| { /* code to benchmark */ }));`
`}`
`criterion_group!(benches, bench_my_function);`
`criterion_main!(benches);`

After setup, you can run your benchmarks using `cargo bench` or, for optimal results, `cargo bench --release`.

Arguments can be passed to the benchmark executable using `--` after all `cargo bench` options. This is especially useful for controlling the behavior of the benchmarking harness itself.

For example, with Criterion.rs, you can use these arguments:
1. Prevent HTML reports: `cargo bench -- --noplot`
2. Compare against a baseline: `cargo bench -- --baseline old_version`
3. Run only a specific benchmark function: `cargo bench -- --test my_benchmark_name`
These arguments vary depending on the chosen benchmarking crate.

Cargo, the Rust package manager and build system, has been an integral part of the Rust programming language since its early development. The `bench` subcommand was introduced to provide a standardized way of measuring code performance, reflecting Rust's strong emphasis on performance. While early Rust versions included a basic benchmarking capability via the `test` crate, the ecosystem quickly gravitated towards more sophisticated external crates like Criterion.rs due to their advanced statistical analysis, plotting capabilities, and robust setup for reliable performance measurement. `cargo bench` has remained the consistent interface for invoking these benchmarks, adapting to and integrating with the evolving best practices of the Rust performance community.

cargo(1), cargo build(1), cargo run(1), cargo test(1)