kfp

kubeflow provides machine learning toolkits on Kubernetes

TLDR

Install Kubeflow on cluster

$ kfctl apply -f [kfctl_config.yaml]

Deploy using manifest

$ kubectl apply -k [manifests/]

Check Kubeflow pods

$ kubectl get pods -n kubeflow

Port-forward to dashboard

$ kubectl port-forward svc/istio-ingressgateway -n istio-system [8080]:80

Submit a pipeline run

$ kfp run submit -e [experiment] -r [run-name] -p [pipeline.yaml]

List pipelines

$ kfp pipeline list

Upload a pipeline

$ kfp pipeline upload -p [pipeline-name] [pipeline.yaml]

Create experiment

$ kfp experiment create -n [experiment-name]

SYNOPSIS

kfctl apply|delete|build [options]
kfp command [options]

Kubeflow provides machine learning toolkits on Kubernetes. kfctl deploys Kubeflow components; kfp manages ML pipelines.
Installation uses kfctl with platform-specific configurations for GKE, EKS, AKS, or vanilla Kubernetes. Modern installations often use kustomize manifests with kubectl apply.
Kubeflow Pipelines (kfp) orchestrate ML workflows as DAGs. Pipelines define components, inputs, outputs, and dependencies. Submit compiled pipelines (YAML/JSON) to the Pipelines service.
The central dashboard provides access to Notebooks (Jupyter), Pipelines, Katib (hyperparameter tuning), KServe (model serving), and other components.
Experiments organize runs. Each run executes a pipeline version with specific parameters. The UI shows run status, logs, metrics, and artifacts.

PARAMETERS

-f, --file config

Configuration file path.

-e, --experiment name

Experiment name or ID.

-r, --run-name name

Name for the run.

-p, --pipeline file

Pipeline file or ID.

-n, --namespace namespace

Kubernetes namespace.

--endpoint url

KFP API endpoint.

KFCTL COMMANDS

apply -f config

Deploy Kubeflow using configuration.

delete -f config

Remove Kubeflow deployment.

build -f config

Generate Kubeflow manifests.

KFP COMMANDS

run submit|list|get

Manage pipeline runs.

pipeline upload|list|get|delete

Manage pipelines.

experiment create|list|get|delete

Manage experiments.

component build

Build component from Python function.

diagnose_me

Diagnose KFP installation.

CAVEATS

Resource-intensive; requires substantial cluster capacity. Installation complexity varies by platform. Component versions may have compatibility issues. Istio is typically required. Multi-user mode needs additional configuration.

HISTORY

Kubeflow was started at Google in 2017 as a way to run TensorFlow on Kubernetes. Open-sourced in 2018, it expanded to support the full ML lifecycle. Version 1.0 released in 2020. The project is now a CNCF sandbox project. Components have modularized, allowing selective installation. KServe (formerly KFServing) became its own project.

kfp