Building blocks fully optimized for model building and training, serving and calculating optimization options
Making it easy to try numerous ideas and techniques and manage various experiments at scale
Reusing components and pipelines to quickly cobble together end-to-end solutions without re-building
Scheduling and Triggering
Scheduling experiments automatically supporting various triggering options
Visibility and Monitoring
Working on a friendly interface to track experiments and model usage and compare different runs
Hyper Parameter Tuning
Running the same experiment with different parameters and easily tracking results
Kubeflow in general, is an open source Kubernetes-native platform for developing, orchestrating, deploying and running scalable and portable ML workloads. Kubeflow Pipelines is a new component from Kubeflow that helps you compose, deploy and manage end-to-end (optionally hybrid) machine learning workflows with UI and a set of services. Users write their own code or build from a large set of pre-defined components and algorithms contributed by companies like Google, Amazon, Microsoft, IBM, NVIDIA, Iguazio, etc.
Once a workflow is in place, you can run it once, at scheduled intervals, or trigger it automatically. The pipelines, experiments and runs are managed, and their results are stored and versioned. Pipelines solve the major problem of reproducing and explaining ML models. It also means you can visually compare between runs and store versioned input and output artifacts in various object/file repositories.
To enable scalability, KubeFlow Pipelines orchestrate various horizontal-scaling and GPU accelerated data and ML frameworks. A single logical pipeline step may run on a dozen parallel instances of TensorFlow, Spark, or Nuclio (Iguazio’s serverless functions). KubeFlow Pipelines also have components which map existing cloud services, to submit a logical task which runs on managed services such as Amazon EMR, Sagemaker, data services and more.