hflow

hflow [-v] [--version] command [options]hflow ingest [uris...] [--profile full|metadatabackfill|relabel] [**--online**] [**--all-stages**] [**--pipeline** file] [**--bundle-dir** dir] [**--airflow-url** url] [**--dag-id** id_]hflow curate [sql] [--sql-file path] [--catalog dir] [-o path] [--dry-run]hflow up [--pipeline file] [--data-root path|url] [--bundle-dir dir] [--api-port port]hflow import lerobot --repo owner/name --output-dir dir [--revision rev] [--camera key] [--episode-index n]

hflow is the command-line interface for HFlow, an open-source SDK for multimodal robotics and physical-AI data pipelines from Hebbian Robotics. Recordings enter as standard MCAP episodes (ROS 2 bags, LeRobot Dataset v3 imports, or other converters). The CLI wraps the same Python library used by `hflow.App`: quality checks, transforms, labels, and enrichments stay ordinary Python functions; hflow handles orchestration, provenance, a Parquet catalog, and curation.A typical local loop does not need Docker. Define a pipeline (`hflow.App(...)` with `@app.check` / enrichments), iterate with `app.test("episode.mcap")`, then ingest episode paths relative to the data root. With no Compose bundle and no $HFLOW_AIRFLOW_URL, ingest runs the stage graph in this process. curate (or dataset create) cuts a version-pinned manifest from DuckDB SQL over the catalog. doctor checks that written files follow HFlow's canonical MCAP convention: complete summary, topic-group chunking, in-band H.264 cameras, and `provenance/v1` stamps (`schemaversion`, `pipelineversion`).When you want a scheduler, up renders an Airflow 3 LocalExecutor stack (not a pip dependency of HFlow) under `<data-root>/runtime` and ingest triggers the master DAG over Airflow's REST API. deploy emits the same DAG bundle for an Airflow 3 you already operate. serve is a workspace HTTP API over the data root; it does not process episodes.Subcommands share a three-value exit convention: 0 success, 1 ran and found something to report (non-conforming doctor files, ingest errors, stale --exit-code hits, up started then failed), 2 bad input and nothing useful happened.

-v, --verbose

Enable verbose logging on stderr. Must appear before the subcommand.

Print `hflow <version>` and exit.

Pipeline file, optionally with the App variable name. Defaults to `hflow.toml`'s `pipeline`, else `./pipeline.py`. Importing the file executes it.

Workspace root: a host directory, or `gs://`, `s3://`, `az://`. Default: $HFLOW_DATA_ROOT, else `hflow.toml`'s `data_root`, else `./data`.

Catalog directory or object-store prefix (default `<data-root>/catalog`). catalog ui refuses bucket URLs.

Manifest path for curate (default `<data-root>/manifest.parquet`) or snapshot directory for export snapshot.

curate: run the query and report row count and coverage without writing a manifest.

ingest stage set. full (default) enables every stage; metadata_backfill re-runs checks and catalog registration; relabel re-runs enrichments only.

ingest latency-first lane: one immediate batch, no bin-packing or stagger. For per-episode runs as files land.

ingest in-process: run every stage of --profile even when the catalog already records current step versions. Use when artifacts were deleted out from under recorded rows.

Rendered Compose bundle. Default `<data-root>/runtime` (or `./runtime` for bucket roots). An explicit bundle keeps ingest/status local even if $HFLOW_AIRFLOW_URL is set.

Remote Airflow API base URL (or $HFLOW_AIRFLOW_URL). Credentials come from the environment only.

Master ingest DAG id on a remote runtime (or $HFLOW_AIRFLOW_DAG_ID).

Hugging Face dataset repository for import lerobot (required).

Branch, tag, or commit to resolve (default main).

Video feature to import; repeatable (default `observation.image`).

Zero-based episode to import (default: every episode).

Read the curate SELECT from a file instead of the command line.

stale: exit 1 when at least one stale episode is found (like `git diff --exit-code`).

Host port for a new bundle's Airflow API (default 8080). An existing `.env` is never rewritten.

down: also remove named volumes (full reset).

Do not open a browser after catalog ui or serve.

serve: refuse manifest pinning, saved-query edits, and run triggering.

export snapshot: keep artifact URIs, or copy files under `assets/` (default references).

hflow.toml

Optional project file in this directory or an ancestor. Typical keys: `data_root = "./data"` and `pipeline = "pipeline.py"`. Relative paths resolve against the file's directory. Runtime URLs and credentials are not stored here.

Overrides the data root. Wins over `hflow.toml` so a shell or control plane can point at another workspace without editing the repo. Resolution order for roots: flag, then this variable, then nearest `hflow.toml`, then `./data`.

Remote runtime addressing for ingest and status. An explicit --bundle-dir stays local; an explicit --airflow-url wins the other way.

Remote Airflow credentials. Never accepted as flags (process listings and shell history). A bearer token wins over username/password.

Use these binaries instead of the checksum-verified ffmpeg/ffprobe HFlow downloads into the user cache on first video operation (Linux x86_64/aarch64).

Per-alias endpoint overrides for pipelines that declare `endpoints={...}` (alias uppercased, non-alphanumerics as `_`). Captured at up render time and forwarded into containers by name only.

Worker-local cache for bucket deployments that spool object-store objects through an etag-validated mirror.

ingest

Submit episode URIs (relative to the data root) to the master ingest DAG, or process them in this process when no runtime is addressed. Default profile full runs sync, meta, labels, and media.

Run a SELECT over catalog views and write `manifest.parquet` (or report coverage with --dry-run). Pass exactly one of a SQL string or --sql-file.

Start DuckDB's local browser UI over the Parquet catalog (default port 4213). Requires a local catalog directory, not a bucket URL. Starts even when the catalog is empty.

Write an immutable version-pinned manifest of every episode the current pipeline stands behind, plus a JSON provenance sidecar. Importing the pipeline executes the file.

Import a Hugging Face LeRobot Dataset v3 repository as canonical MCAP episodes. Resolves the revision to a commit and records it as provenance.

Write selected catalog tables (and optional artifact media) to a local directory of Parquet files.

Print source URIs of episodes whose latest cataloged run predates the current pipeline (and format) version, one per line on stdout. Summary goes to stderr.

Validate one or more MCAP files against the canonical-episode convention (container, stamps, chunk groups, in-band H.264). Reports each file in argument order.

Import the pipeline and print its step names, versions, gates, and endpoints as JSON.

Render a Docker Compose bundle for Apache Airflow 3 and start it detached. First start pulls images (~2 GB) and builds the task venv.

Stop the Compose runtime. --volumes also drops the Postgres metadata DB and user-venv volumes; the data root is never removed.

Print API URL, Airflow component health, and Compose service state (or REST-only health for a remote runtime).

Emit DAG files, `user/` copies, and a `DEPLOY.md` for an existing Airflow 3 deployment (Astronomer, MWAA, Cloud Composer, self-managed). Calls no platform API.

Serve the workspace over HTTP (REST over the catalog, optional UI). Requires the separate hflow-server package. Distinct from up, which starts the processing runtime.

HFlow is pre-v1. Canonical MCAP episodes are derived artifacts, not a byte-stable source of record; version stamps make regenerations visible. Requires Python 3.11+.The Hebbian Robotics project on PyPI starts at 0.2.0. Earlier 0.1.x releases under the same name belonged to an unrelated, inactive project before the name was transferred.ingest URIs must be relative to the data root. Absolute host paths and `../` escapes are rejected before anything is triggered (they would not resolve inside the runtime mount).catalog ui needs a local directory. Bucket catalogs (`gs://`, `s3://`, `az://`) are refused. The first launch may download DuckDB's `ui` extension.serve is not in the core wheel: install `hflow-server` separately (`uv add hflow-server`). Binding --host past loopback exposes the corpus.up is explicit and heavy: the first start pulls ~2 GB of images and builds a task venv. Ingest never starts Airflow on your behalf. Windows Airflow support is WSL2 only; keep the data root on the Linux filesystem.Importing a pipeline (manifest, dataset create, stale, in-process ingest, serve's pipeline page) executes the file. Run those commands in the pipeline's own environment. Guard `app.run()` with `if _name == "main_"` so Airflow workers do not re-enter it on import.doctor does not currently classify H.264 picture coding types (B-frames) and does not reject non-VCL NAL units before the first access-unit delimiter; a clean report is not proof of those two constraints.

HFlow is developed by Hebbian Robotics (Y Combinator S26) as open-source data-quality infrastructure for physical AI. The CLI entry point is `hflow.cli:main` (PyPI package hflow, Apache-2.0). Version 0.2.4 documents the current subcommand set: curate, catalog, dataset, import, export, stale, doctor, manifest, up/down/ingest/status, deploy, and serve.

python(1), python3(1), uv(1), pip(1), duckdb(1), ffmpeg(1), airflow(1), docker(1), docker-compose(1)

Source code · Homepage · Documentation