koboldcpp

Launch with a GGUF model file

$ koboldcpp --model [path/to/model.gguf]

$ koboldcpp --model [path/to/model.gguf] --usecuda --gpulayers [35]

$ koboldcpp --model [path/to/model.gguf] --usevulkan --gpulayers [35] --port [8080]

$ koboldcpp --model [path/to/model.gguf] --prompt "[What is the meaning of life?]"

$ koboldcpp --model [path/to/model.gguf] --cli

$ koboldcpp --config [path/to/config.kcpps]

koboldcpp [options] [--model modelpath_]

koboldcpp is a self-contained AI text generation server that runs large language models locally. Built on top of llama.cpp, it provides a bundled web UI (KoboldAI Lite) and supports all GGML and GGUF model formats. It requires no external dependencies and runs as a single executable.
The server exposes an API compatible with KoboldAI and OpenAI formats, making it usable with a wide range of frontends and applications. It supports CPU inference as well as GPU acceleration through CUDA (NVIDIA), Vulkan (AMD/NVIDIA), and Metal (Apple Silicon).
Beyond text generation, koboldcpp supports image generation (Stable Diffusion), speech recognition (Whisper), and text-to-speech, all within the same executable. The bundled web UI offers multiple interaction modes including chat, instruct, adventure, and story writing.

--model path

Specify the GGUF/GGML model file to load

Load a .kcpps configuration file

Enable NVIDIA CUDA GPU acceleration

Enable Vulkan GPU acceleration (AMD/NVIDIA)

Number of model layers to offload to GPU

Set CPU thread count for inference

Set maximum context length in tokens

Change server port (default: 5001)

Bind to a specific IP address

Enable multiuser mode with n concurrent slots

Require API authentication with the given key

Launch interactive command-line interface without starting a server

Run a single prompt, print output, and exit

Run performance benchmarking mode

Enable flash attention for improved performance

Enable smart context handling to reduce reprocessing

Enable memory-mapped file I/O for model loading

Force model to remain in RAM (prevent swapping)

Enable SSL for HTTPS connections

Enable remote tunnel access for sharing the server

Load a Stable Diffusion model for image generation

Compatibility mode for older CPUs without AVX2

Show the GUI launcher even when command-line flags are used

Display all available commands

When launched without arguments, koboldcpp opens a GUI launcher for interactive configuration. Settings can be saved to and loaded from .kcpps configuration files. Command-line flags override GUI settings when both are used.
Key configuration considerations include GPU layer offloading (more layers on GPU means faster inference but requires more VRAM), context size (larger contexts use more memory), and thread count (typically set to the number of physical CPU cores).

Model files can be very large (several GB to over 100 GB) and require significant RAM or VRAM. GPU acceleration requires appropriate drivers and hardware support. Performance varies significantly based on model size, quantization level, and available hardware. The Vulkan backend is more broadly compatible but generally slower than CUDA on NVIDIA hardware. Flash attention requires compatible model architectures.

KoboldCpp was created by a developer known as LostRuins (alias Concedo) and first released on March 16, 2023 as a fork of llama.cpp combined with the KoboldAI interface. It was designed to provide a simple, self-contained way to run large language models locally without complex setup. The project grew rapidly alongside the open-source LLM movement, continuously adding features like multi-modal support, GPU backends, and image generation capabilities. It is licensed under AGPL-3.0.

llama(1), ollama(1)