janas-bench

Measure how a Janas-LLM model runs on this machine

TLDR

List catalog models and pick one

$ janas-bench

Benchmark a catalog name (downloaded if missing)

$ janas-bench [qwen3-4b]

Benchmark a converted .jns file

$ janas-bench [models/qwen3-next.jns]

Match a llama-bench-style long prompt

$ janas-bench [qwen3-4b] --prompt [1024] --gen [16]

Average three rounds (the default)

$ janas-bench [qwen3-4b] --rounds [3]

Skip the model's prediction file

$ janas-bench [qwen3-4b] --no-mtp

janas-bench measures what this machine does with a Janas-LLM model and writes a report. It tries the configurations the engine can use (thread counts, which cores, with and without the GPU) on real passes of the model, keeps the fastest, and stores them in `~/.cache/janas` so janas-chat and the other tools start already tuned.model is taken as janas-chat takes it: a `.jns` file, a catalog name (`qwen3-4b`), downloaded after asking if it is not there yet, or nothing, and then the list to choose from. The model's prediction file comes with it unless --no-mtp.Default work is a prompt of 256 tokens then 64 written, averaged over 3 rounds with the order of configurations alternating so warmup and cache fill weigh on each setting alike. Prefill and decode are measured greedily without drafts. Drafts from the prediction block are measured as a chat uses them: an ordinary English question, greedily and sampled the way the chat samples, with the share of drafts that survived.Close other programs first. The report is `janas-bench-<date>.txt` in the current directory and already carries the CPU name, core layout, GPU if any, RAM, and the speed of every configuration tried. Send that file with hardware feedback.

PARAMETERS

model

A `.jns` path or catalog name. None: list the catalog and ask.

--models dir

Where models live (sets JANAS_MODELS for this run).

--mtp file

Multi-token prediction file. Without it, the matching file beside the model.

--no-mtp

No prediction file.

--prompt tokens

Prefill length (default 256).

--gen tokens

Tokens written after the prompt (default 64).

--ctx tokens

Context length (default 4096). Prompt plus generation must fit.

--cache GiB

Memory for streamed experts.

--reserve GiB

Memory left to other programs when the cache is automatic.

--rounds n

How many times each configuration is measured (default 3, capped at 1000).

--settle seconds

After warmup, wait up to this long for background expert-cache filling to finish, so later rounds match a chat after its first minutes.

-h, --help

Print usage and exit 2.

CAVEATS

The engine is deliberately hard on the machine: it fills free memory and reads the disk at full speed. Run it on an idle machine; a busy desktop is part of what is measured. Prompt plus twice generation plus 64 must fit --ctx. --mtp and --no-mtp together is an error. Figures vary a few percent between runs; a first run is slower while the cache fills. Developed on one Linux x86-64 laptop; reports from other CPUs, discrete GPUs, slower disks, and 16 GB machines are what the project asks for most.

HISTORY

janas-bench ships with Janas, written by Maurizio "camauri" Cammalleri (first public release September 2026, GNU GPL 3 or later). Until 26 September 2026 the draft column measured a repeating paragraph that a model copies, which inflated "with MTP" rates; later builds measure a real chat question instead.

RESOURCES

Source code · Documentation