slotstream

slotstream subcommand [options]

slotstream is a Swift CLI that runs the 104 GB 4-bit Qwen3.8-Flash-Next mixture-of-experts model on Apple Silicon Macs that cannot hold the whole checkpoint in RAM. The dense trunk (~3.8 GB) stays resident; 68 GB of routed experts (512 per layer, 10 active per token) and a 32 GB n-gram table are read from SSD into a fixed pool of cache slots shared across all 48 layers.serve listens on port 11434 and implements the chat/generate subset used by Ollama clients and OpenAI SDKs (streaming, CORS, and common sampling fields). The advertised model name is qwen3.8-flash-next:4bit. Unsupported features (tools, images, JSON-schema output, logprobs, other model names) return HTTP 400 instead of being ignored.With no size flags, auto picks the lowest of 33 GB, 70% of RAM, and the Metal working-set limit, then stays elastic: every 15 s it may shrink or grow the cache between requests. Greedy decoding is byte-identical across cache sizes. Follow-up turns reuse prefix state so time-to-first-token stays flat; --no-prefix-cache disables that.pull downloads 103.8 GB in 24 files from Hugging Face, resumes on interrupt, and checks sha256 hashes compiled into the binary. serve and run offer the same download on first use if a terminal is attached.

run

Generate from a prompt. --prompt TEXT (default: a built-in sample), --max-tokens N (default 128; ≤0 means as many as allowed), --greedy (deterministic), --raw (no chat template), --think (thinking mode).

Start an Ollama-compatible HTTP server. --port N (default 11434), --max-context N (default 32768), --no-prefix-cache (re-prefill every request), --no-elastic (pin the expert cache at startup size).

Download the pinned 4-bit weights (only qwen3.8-flash-next:4bit in v0). --dir PATH (default `~/.slotstream/models/...`), --connections N, --verify (re-hash an existing copy; download nothing).

Print device RAM, the memory plan your flags would produce, estimated decode speed, and whether the weights fit on disk. --json, --sim-ram GB, --sim-working-set GB, --sim-available GB.

Model name or directory (default: the pinned Qwen3.8-Flash-Next 4-bit build).

Total memory the process may use, in GB (minimum 8.1). First of the three size knobs that wins.

Cache N of 512 routed experts per layer (pool ≈ N × 0.133 GB). Takes precedence over --memory-gb and --pool-gb.

Raw expert-pool size in GB. Beats --memory-gb; loses to --experts-per-layer.

Auto sizing only: largest share of RAM auto may target (default 70). Cannot raise the target past the ~33 GB decode knee.

Requires Apple Silicon and macOS 14+. It does not run on Linux or Intel Macs. You need about 110 GB free disk for the weights. Performance numbers in the upstream README were measured on one M5 Pro with 48 GB; smaller RAM tiers are estimated. Long prompts are slow to start (prefill is paid up front). Prefix-cache reuse is not bit-identical to a full re-prefill, so greedy replies can differ on near-tied tokens. One model process at a time. macOS 14 and 15 have had the installer exercised, not a full runtime campaign.

slotstream was published as a Show HN project by carloslfu in 2026. It is a single Swift binary using MLX, written to run Qwen3.8-Flash-Next by streaming MoE experts rather than mmap'ing the full 104 GB checkpoint.

ollama(1), llama.cpp(1), llamafile(1)

Source code