prefect

Start Prefect server

$ prefect server start

$ prefect deploy [flow.py:flow_name]

$ prefect flow-run create [flow-name]

$ prefect deployment ls

$ prefect worker start -p [work-pool]

$ prefect work-pool create [pool-name] -t [process]

$ prefect flow-run ls

$ prefect cloud login

prefect command [options]

prefect is the CLI for Prefect, a Python workflow orchestration framework. It manages deployments, workers, and infrastructure for running data pipelines.
Flows are Python functions decorated with @flow. prefect deploy packages flows for remote execution with schedules, parameters, and infrastructure settings.
Workers poll work pools and execute flow runs. prefect worker start launches a worker for a specific pool. Work pool types define execution environment: process (local), Docker, or Kubernetes.
prefect server start runs the API server and UI locally. For production, use Prefect Cloud or self-hosted with PostgreSQL. The UI shows flow runs, logs, and task states.
Blocks store configuration like credentials, storage locations, and infrastructure templates. Register blocks with block register and reference them in deployments.

-p, --pool name

Work pool name.

Work pool type: process, docker, kubernetes.

Deployment or resource name.

Cron schedule for deployment.

Interval schedule for deployment.

Limit results.

server start

Start local Prefect server.

Deploy a flow.

Manage flow runs.

Manage deployments.

Start a worker process.

Manage work pools.

Manage storage and infrastructure blocks.

Prefect Cloud authentication.

Manage configuration profiles.

Configuration management.

Prefect 2.x is significantly different from Prefect 1.x. Workers replace agents from 1.x. Deployments require accessible code (Git, Docker, storage). Cloud features require account.

Prefect was founded by Jeremiah Lowin in 2018 as a modern alternative to Airflow. Prefect 1.0 (2019) introduced the Hybrid model with local execution. Prefect 2.0 (2022) was a complete rewrite emphasizing simplicity with native Python code as workflows. The company offers Prefect Cloud for managed orchestration. Prefect has grown popular for data engineering and ML pipelines.

dagster(1), airflow(1), luigi(1), argo(1)