litert-lm
run on-device LiteRT-LM language models from the command line
TLDR
SYNOPSIS
litert-lm command [options]
DESCRIPTION
litert-lm is the CLI for Google's LiteRT-LM stack: on-device LLM inference with LiteRT, used in Chrome, Chromebook Plus, and Pixel Watch. It runs `.litertlm` packages (Gemma and others) on CPU or GPU, with optional vision/audio attachments and Python tool loops.Install from PyPI (`uv tool install litert-lm` or `pip`) or from GitHub releases. GPU needs working drivers (Vulkan on Linux/Windows, Metal on macOS). MTP (--enable-speculative-decoding) only works when the model includes a drafter.
PARAMETERS
--from-huggingface-repo owner/name
Download the named file from that repository.--prompt text
One-shot prompt instead of a REPL.--backend cpu|gpu
Inference backend.--enable-speculative-decoding true
Multi-token prediction when the model ships a drafter (GPU).--vision-backend / --audio-backend cpu|gpu
Required when using image or audio --attachment (repeatable).--preset file.py
Python file defining `tools` and optional `system_instruction` for function calling.--port n
Listen port for serve.-h, --help
Help for the tool or a subcommand (`litert-lm run --help`).
COMMANDS
run [model | id] [options]
Interactive chat or a single --prompt.serve [options]
OpenAI-compatible API server.benchmark model
Measure decode/prefill performance.list
List imported models.
CAVEATS
Not every `.litertlm` file supports MTP, GPU, or multimodality. Hugging Face downloads need network and accept the model license. serve is a local API; do not bind it on an untrusted network without your own access control.
HISTORY
LiteRT-LM CLI shipped around the Gemma 4 on-device releases (v0.10+). Later versions added Windows GPU, `serve`, Android, and C API prebuilts.
