llama-cli

llama-cli -m model.gguf [options]llama-cli -hf user/model[:quant] [options]

llama-cli is the command-line inference tool of the llama.cpp project. It loads a model in the GGUF format and generates text either interactively, as a back-and-forth chat, or in a single non-interactive completion. It runs entirely on local hardware, using the CPU, a GPU (CUDA, Metal, Vulkan, ROCm), or a mix of both.By default llama-cli enters conversation mode when the model ships with a chat template, applying that template automatically and exposing an interactive prompt. The -no-cnv flag disables this and treats the input strictly as a raw completion prompt. Models can be supplied as a local file with -m, downloaded on demand from Hugging Face with -hf, or pulled from a Docker repository with -dr.The tool exposes the full sampler stack (temperature, top-k, top-p, min-p, mirostat, DRY, and more), context management options such as context size and context shifting, and hardware controls for thread count and GPU layer offloading. It shares its argument parser with the other llama.cpp tools like llama-server, so most flags behave identically across them.

-m, --model FNAME

Path to the GGUF model file to load.

Download the model from a Hugging Face repository; quant defaults to Q4KM.

Prompt to start generation with. For a system message use -sys.

Read the prompt from a file.

System prompt to use with the model, when the chat template supports one.

Size of the prompt context (default: 0, loaded from the model).

Number of tokens to predict (default: -1, -1 = infinity).

Maximum number of layers to store in VRAM. Accepts an exact number, auto, or all.

Number of CPU threads to use during generation.

Force conversation mode on or off (auto-enabled when a chat template is present).

Sampling temperature (default: 0.80).

Colorize output to distinguish prompt and input from generated text.

Run in interactive mode, returning control on a reverse prompt.

Halt generation at PROMPT and return control in interactive mode.

llama-cli only reads GGUF files; older GGML model files must be converted or re-downloaded. Performance and the achievable context size depend heavily on quantization and available VRAM: setting -ngl higher than your GPU memory allows causes a load failure or fallback to CPU. The single binary was historically named main; older guides referring to ./main describe the same program.

llama.cpp was created by Georgi Gerganov in March 2023 as a plain C/C++ port of Meta's LLaMA model, aimed at running inference efficiently on consumer hardware without a GPU. The example program that became llama-cli was originally called main; it was renamed to llama-cli in 2024 when the project standardized its tool names. Development continues under the ggml-org organization with frequent releases.

llama.cpp(1), llamafile(1), ollama(1), llm(1)

Source code · Documentation