embedflow

embedflow [--version] command [options]

embedflow is the command-line interface for EmbedFlow, a Python toolkit that upgrades an embedding model over an existing vector index without waiting for a full corpus re-embed. Queries are encoded with the source model, the existing index returns a candidate neighborhood, and those candidates are scored with the target model while target document vectors are materialized in the background.Search responses expose a warmth state: COLD (no target vectors yet), PARTIAL (only currently cached candidates are ranked), or WARM (the candidate set is fully materialized). Once the same candidate set is warm, target scoring is deterministic.analyze is the no-target-index workflow. It probes finite-tail behavior with unlabeled queries and reports a T2-v1 diagnostic of SAFE, EXPAND, or UNSAFE_OR_UNCERTAIN, plus a recommended initial candidate depth. Treat SAFE as an empirical deployment signal, not a guarantee. evaluate is the labelled workflow: it needs qrels and can compute source quality, native target quality, target-within-source-candidates quality, candidate gap `G(K)`, containment, and observed migration depth.The CLI is registered as the `embedflow` console script (`embedflow.cli:main`). Core extras install FAISS, Qdrant, Hugging Face model runtimes, and the FastAPI dashboard (`pip install "embedflow[faiss,dashboard]"` or `"embedflow[all]"`). Requires Python 3.10+. Licensed AGPL-3.0-only.

--config file

Path to an EmbedFlow YAML file. Relative paths inside the file are resolved against that file's directory. Default for most commands: `embedflow.yaml`.

JSONL document store. Rows must contain the configured ID and text fields (defaults: `id`, `text`).

Existing FAISS index path, or a Qdrant URL when `--backend qdrant`.

Vector index backend. Default: `faiss`.

Legacy embedding model ID or local path.

Target embedding model ID or local path.

JSONL unlabeled probe queries (`query` or `text` field).

Override the configured model device. `gpu` is accepted and mapped to `cuda`.

Directory of staged model snapshots. Known IDs are resolved to subdirectories when present.

Where analysis or evaluation writes reports and generated YAML.

Source-index candidate pool size used for target scoring (`migrate` default: 50). `analyze` can set this to `auto`.

Number of results returned by search. Default: 10.

Bind address for serve, demo, migrate, and real-demo. Defaults: `127.0.0.1` and `8000`.

Write/validate config or build the demo without starting the HTTP service.

Emit machine-readable JSON from doctor, economics, and registry subcommands.

Print the package version and exit.

Most commands take --config. Relative paths are resolved against the YAML file's directory so the same file can move between machines.

$ source:
  model: sentence-transformers/all-MiniLM-L6-v2
  device: cuda
target:
  model: Qwen/Qwen3-Embedding-0.6B
  device: cuda
index:
  backend: faiss
  path: ./legacy.index
  ids: ./legacy.index.ids.json
  metric: cosine
documents:
  path: ./documents.jsonl
  id_field: id
  text_field: text
probe:
  queries: ./probe_queries.jsonl
  kmax: 500
  epsilon: 0.01
migration:
  candidate_depth: auto
  max_sync_misses: 4
  background_batch_size: 32
cache:
  path: ./embedflow_cache
state_path: ./embedflow_state.json

init

Create or validate an EmbedFlow YAML configuration.

Connect an existing FAISS or Qdrant index and start progressive migration.

Run the no-target-index finite-tail (T2-v1) diagnostic and write a reusable config.

Compute labelled candidate-gap metrics against qrels and optional native target rankings.

Start the FastAPI service and dashboard (default `127.0.0.1:8000`).

Search with source-index retrieval and target-model reranking.

Report cache coverage, hit/miss counters, queue depth, and materialization throughput.

Materialize selected target vectors without changing source-index results.

Compare ANN results against an exact/reference source index where supported.

Materialize all target vectors and build a native target index.

Project backfill time and GPU cost from supplied corpus size and throughput.

Run the self-contained offline progressive-migration demo.

Run a small MiniLM → Qwen3-0.6B demo with public model snapshots.

Check Python, optional dependencies, paths, dimensions, cache integrity, and contract fingerprints.

Inspect, match, or checksum the packaged migration-evidence registry.

List workload-specific measured latency/throughput profiles.

v0.1 is an alpha release for research and early testing. PARTIAL rankings can differ from fully warm target reranking. ANN fidelity is independent of T2-v1 and stays UNKNOWN until an exact/reference source comparison is supplied. Registry rows are prior evidence with documented contracts; they do not replace evaluation on the current corpus. Latency and cost projections depend on hardware, model runtime, corpus, batch size, and workload. The default `epsilon=0.01` is a stringent study setting, not a universal production threshold. Real-model commands need optional extras (`faiss`, `models`, `dashboard`) and typically a GPU for larger target models.

EmbedFlow was written by Arnav Srivastav and first published as v0.1.0 in 2026 (AGPL-3.0-only). The initial release shipped T2-v1 analysis, FAISS and Qdrant adapters, a persistent target-vector cache, a FastAPI dashboard, and a packaged evidence registry of measured embedding-model transitions.

pip(1), python(1), huggingface-cli(1), uvicorn(1), fastapi(1)

Source code · Homepage · Documentation