Kubeflow Pipelines on Kubernetes with KIND: End-to-End Guide for ML and Cloud Engineers

Introduction

Machine Learning (ML) workflows involve much more than training models. They require reproducible pipelines, scalable infrastructure, experiment tracking, and automated deployments. Kubeflow Pipelines (KFP) provides a Kubernetes-native platform for designing, automating, and managing end-to-end ML workflows.

In this guide, you'll learn how to deploy and use Kubeflow Pipelines on a Kubernetes cluster running with KIND (Kubernetes IN Docker). Whether you're an ML Engineer, Cloud Engineer, or DevOps Engineer, this guide will help you build portable, scalable, and reproducible machine learning pipelines in a local Kubernetes environment.

What is Kubeflow?

Kubeflow is an open-source Machine Learning platform built specifically for Kubernetes. It simplifies the development, orchestration, deployment, and management of machine learning workflows while leveraging Kubernetes for scalability and reliability.

Kubeflow supports the complete machine learning lifecycle—from interactive development and model training to deployment, monitoring, and experiment management.

Key capabilities include:

• Native integration with Kubernetes
• End-to-end machine learning workflow management
• Interactive Jupyter notebooks
• Automated model training and experimentation
• Model serving and deployment
• Reproducible and version-controlled ML pipelines
• Highly scalable and customizable architecture

Why Use Kubeflow?

Managing production-grade machine learning projects involves several challenges beyond model development, including:

• Packaging applications into containers
• Automating ML workflows
• Versioning datasets and models
• Monitoring experiments
• Deploying models reliably
• Scaling workloads across Kubernetes clusters

Kubeflow simplifies these tasks by providing:

• Reusable pipeline components
• Visual pipeline authoring
• Automated workflow execution
• Experiment tracking and comparison
• Kubernetes-native scalability
• Integrated user interface for managing ML workloads

By standardizing ML workflows, Kubeflow enables data scientists, ML engineers, and platform engineers to collaborate more effectively.

Understanding Kubeflow Components

Kubeflow follows a modular architecture, allowing you to install and use only the components required for your machine learning workflows.

Kubeflow Project Components

The major components of Kubeflow include:

What are Kubeflow Pipelines (KFP)?

Kubeflow Pipelines (KFP) is a workflow orchestration platform within Kubeflow that enables you to design, automate, and manage machine learning workflows on Kubernetes.

Instead of manually executing each stage of an ML project, Kubeflow Pipelines organizes the entire workflow into reusable components connected in a defined sequence. This makes ML processes reproducible, scalable, and easier to maintain.

A typical pipeline may include tasks such as data preparation, model training, evaluation, deployment, and monitoring, all executed automatically based on defined dependencies.

Key Concepts

Traditional SDLC vs Machine Learning Workflow

Although Machine Learning projects follow many software engineering principles, the lifecycle differs significantly from a traditional Software Development Life Cycle (SDLC). ML workflows introduce additional stages such as data preparation, feature engineering, model training, experimentation, and continuous monitoring.

The following diagrams illustrate the differences between a traditional SDLC and a typical Machine Learning workflow.

Example: Restaurant Demand Prediction

Imagine a restaurant wants to predict how many meals will be served next week so it can better manage inventory and staffing.

The workflow could include the following stages:

1. Data Collection – Gather historical sales data, weather information, and other relevant records.
2. Data Cleaning – Remove missing values, correct inconsistencies, and prepare the dataset.
3. Model Training – Train a machine learning model using the processed data.
4. Model Evaluation – Measure the model's prediction accuracy using validation data.
5. Model Deployment – Deploy the trained model as an API for generating predictions.
6. Monitoring – Continuously compare predictions with actual sales and retrain the model when necessary.

Example: Office Attendance Prediction

An organization wants to estimate how many employees will work from the office next week to optimize workspace planning.

A typical ML workflow would involve:

1. Data Collection – Collect employee attendance records from the previous six months.
2. Data Analysis – Identify trends such as weekdays, seasonal patterns, and attendance frequency.
3. Feature Engineering – Incorporate additional features such as holidays, project deadlines, weekdays, and special events.
4. Model Training – Train a prediction model using the prepared dataset.
5. Model Evaluation – Validate the model's performance using historical data.
6. Deployment – Publish the model so managers can access attendance predictions.
7. Monitoring – Continuously evaluate prediction accuracy and update the model as new data becomes available.

Understanding the Traditional Software Development Life Cycle (SDLC)

The Software Development Life Cycle (SDLC) is a structured process used to design, develop, test, deploy, and maintain software applications. Each phase has a specific purpose and helps ensure that software is delivered with high quality and reliability.

A typical SDLC consists of the following stages:

Example: Building an Employee Leave Management System

Consider an organization developing a leave request application.

The SDLC process would typically look like this:

1. Planning – Identify user roles such as employees, managers, and HR administrators.
2. Analysis – Define business rules including leave balances, approval workflows, and holiday calendars.
3. Design – Create UI wireframes, database schema, and application architecture.
4. Implementation – Develop the frontend using React and the backend using Node.js.
5. Testing – Perform unit, integration, and user acceptance testing.
6. Deployment – Deploy the application on Google Cloud Platform (GCP).
7. Maintenance – Enhance the application by adding features such as email notifications and reporting dashboards.

Understanding the Machine Learning Lifecycle

Unlike traditional software development, Machine Learning projects are data-driven and involve continuous experimentation. Instead of simply writing business logic, ML workflows focus on preparing data, training models, evaluating performance, and continuously improving predictions.

A typical Machine Learning lifecycle includes the following stages:

Example: Predicting Office Attendance

Suppose an organization wants to predict the number of employees expected to work from the office each week.

A typical ML workflow would be:

1. Problem Definition – Predict weekly employee attendance.
2. Data Collection – Gather historical attendance records.
3. Exploratory Data Analysis – Analyze daily, weekly, and seasonal attendance patterns.
4. Feature Engineering – Add features such as weekdays, holidays, weather conditions, and scheduled meetings.
5. Model Training – Train a regression model using the prepared dataset.
6. Model Evaluation – Evaluate prediction accuracy using historical data.
7. Model Deployment – Publish the trained model as a prediction service for HR teams.
8. Monitoring – Compare predicted attendance with actual attendance and retrain the model as new data becomes available.

Additional Kubeflow Features

Kubeflow provides several advanced capabilities that make it suitable for production-grade Machine Learning workflows. These features improve scalability, reproducibility, and flexibility across different ML projects.

Define Input and Output Dataset Types

Kubeflow components can explicitly define the format of their inputs and outputs. This improves interoperability between pipeline components and helps validate data flowing through the workflow.

Example

@dsl.component  
def clean_data(input_csv: str) -> str:  
    # Read CSV and output cleaned JSON  
    return output_json  

In this example, the component accepts a CSV file as input and produces cleaned JSON data that can be consumed by downstream pipeline components.

Enable GPU Acceleration

Training deep learning models often requires GPU acceleration. Kubeflow allows pipeline components to request GPU resources directly from Kubernetes.

Example

resources:  
  limits:  
    nvidia.com/gpu: 1  

Add this configuration to your pod specification or component definition to enable GPU-based execution.

Use Different Base Images

Each Kubeflow component can use a different container image depending on the required runtime environment.

Common base images include:

• python:3.8-slim
• tensorflow/tensorflow:latest-gpu
• huggingface/transformers
• ubuntu:20.04

Example

@dsl.component(base_image="tensorflow/tensorflow:latest-gpu")  
def train_dl_model():  
    ...  

Using specialized base images reduces setup time by providing pre-installed machine learning frameworks and dependencies.

Build and Use Custom Docker Images

In some scenarios, the official container images may not include all the required tools, libraries, or operating system packages. Kubeflow allows you to build and use custom Docker images tailored to your project requirements.

For example, a custom image might include:

• GCC compiler
• Make utility
• CUDA libraries
• Specific Python packages
• Custom machine learning frameworks
• Organization-specific dependencies

Custom Docker images provide greater flexibility, ensure consistent execution environments, and improve reproducibility across development, testing, and production environments.

Practical: Build a Custom Docker Image for Kubeflow Pipelines

In this example, you'll create a custom Docker image containing the required machine learning libraries and use it within a Kubeflow Pipeline component. This approach is useful when your pipeline requires additional system packages or Python dependencies that are not available in the default runtime image.

Environment: Ubuntu (WSL)

Set Up a Python Virtual Environment

Create and activate a Python virtual environment before installing the Kubeflow Pipelines SDK.

mkdir kfp && cd kfp

sudo apt update

sudo apt install python3.10-venv

python3 -m venv .kfp

source .kfp/bin/activate

pip install kfp==2.9.0  

Step 1: Create a Custom Docker Image

Create a Dockerfile with the required operating system packages and Python libraries.

FROM python:3.10-slim

RUN apt update && apt install -y gcc g++ make

RUN pip install pandas numpy scikit-learn  

This image includes:

• Python 3.10
• GCC and G++ compilers
• Make utility
• Pandas
• NumPy
• Scikit-learn

These dependencies will be available to every pipeline component that uses this image.

Step 2: Build and Push the Docker Image

Build the custom image locally and push it to Docker Hub.

docker build -t pavan5779/custom-ml-image .

docker push pavan5779/custom-ml-image  

After the image is pushed, it can be pulled by Kubernetes whenever the pipeline executes.

Step 3: Use the Custom Image in a Kubeflow Pipeline

Reference the custom Docker image using the base_image parameter when defining a pipeline component.

from kfp import dsl, compiler  
from typing import NamedTuple

@dsl.component(base_image="pavan5779/custom-ml-image")  
def custom_training_component() -> NamedTuple("Output", [("result", str)]):  
    import pandas as pd  
    from sklearn.linear_model import LinearRegression

    data = pd.DataFrame({"x": [1, 2, 3, 4], "y": [2, 4, 6, 8]})

    model = LinearRegression()  
    model.fit(data[["x"]], data["y"])

    result = model.predict([[5]])[0]

    print(f"Predicted: {result}")

    return (str(result),)

@dsl.pipeline(name="custom-base-ml-pipeline")  
def custom_pipeline():  
    custom_training_component()

if __name__ == "__main__":  
    compiler.Compiler().compile(custom_pipeline, "custom_pipeline.yaml")  

In this example:

• The pipeline uses the custom Docker image as its runtime.
• A simple linear regression model is trained using Scikit-learn.
• The model predicts the output for the value 5.
• The compiled pipeline is saved as custom_pipeline.yaml.

Step 4: Upload and Run the Pipeline

After compiling the pipeline, upload it through the Kubeflow Pipelines web interface.

Navigate to:

Kubeflow Pipelines → Upload Pipeline

Then:

1. Upload custom_pipeline.yaml.
2. Create a new pipeline run.
3. Execute the pipeline.
4. Open the component logs to verify the prediction output.

Expected output:

{  
  "result": "10.0"  
}  

The successful execution confirms that the custom Docker image was pulled correctly and the required ML libraries were available during pipeline execution.

Deploy Kubeflow Pipelines on a KIND Kubernetes Cluster

In this section, you'll set up a local Kubernetes cluster using KIND (Kubernetes IN Docker) and deploy Kubeflow Pipelines (KFP). This lightweight setup is ideal for learning, development, and testing machine learning workflows without requiring a managed Kubernetes cluster.

Step 1: Install Docker on Ubuntu

Install Docker Engine, which is required to run the KIND Kubernetes cluster.

sudo apt-get update

sudo apt-get install ca-certificates curl gnupg

sudo install -m 0755 -d /etc/apt/keyrings

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | \  
  sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg

echo \  
  "deb [arch=\"$(dpkg --print-architecture)\" signed-by=/etc/apt/keyrings/docker.gpg] \  
  https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | \  
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io  

After the installation completes, verify that Docker is installed and running correctly before continuing.

Step 2: Install KIND and Create a Kubernetes Cluster

Download the latest KIND binary, make it executable, and move it to your system path.

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.29.0/kind-linux-amd64

chmod +x ./kind

sudo mv ./kind /usr/local/bin/kind  

Create a Kubernetes cluster named kfp-demo-yt.

kind create cluster --name=kfp-demo-yt  

Once the cluster is created successfully, verify that the Kubernetes control plane is running before proceeding with the Kubeflow installation.

Step 3: Install Kubeflow Pipelines

Deploy the Kubeflow Pipelines components to the KIND cluster using the official manifests.

First, define the Kubeflow Pipelines version.

export PIPELINE_VERSION=2.4.0  

Install the cluster-scoped resources.

kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/cluster-scoped-resources?ref=$PIPELINE_VERSION"  

Wait for the required Custom Resource Definitions (CRDs) to become available.

kubectl wait --for condition=established --timeout=60s crd/applications.app.k8s.io  

Finally, deploy the Kubeflow Pipelines environment.

kubectl apply -k "github.com/kubeflow/pipelines/manifests/kustomize/env/dev?ref=$PIPELINE_VERSION"  

The installation creates the namespaces, controllers, services, and other Kubernetes resources required to run Kubeflow Pipelines locally.

Step 4: Access the Kubeflow Pipelines UI

After installing Kubeflow Pipelines, expose the web interface using Kubernetes port forwarding.

Run the following command:

kubectl port-forward -n kubeflow svc/ml-pipeline-ui 8080:80  

Open your browser and navigate to:

http://127.0.0.1:8080/#/pipelines  

You should now see the Kubeflow Pipelines dashboard, where you can upload, manage, and execute ML pipelines.

Your First Kubeflow Pipeline: Hello World

Let's begin with a simple pipeline that prints a greeting. This example demonstrates the basic structure of a Kubeflow Pipeline, including a component, a pipeline definition, and compilation into a deployable YAML file.

Create a file named hello_pipeline.py.

from kfp import dsl, compiler

@dsl.component  
def say_hello(name: str) -> str:  
    message = f"Hello, {name}!"  
    print(message)  
    return message

@dsl.pipeline(name="hello-world")  
def hello_pipeline(name: str = "Kubeflow"):  
    say_hello(name=name)

if __name__ == "__main__":  
    compiler.Compiler().compile(hello_pipeline, "hello_pipeline.yaml")  

Compile the pipeline to generate the hello_pipeline.yaml specification.

Upload the Pipeline

Open the Kubeflow Pipelines UI and upload the generated hello_pipeline.yaml file.

Create and Run the Pipeline

After uploading the pipeline:

1. Click Create Run.
2. Configure the pipeline parameters if required.
3. Start the execution.

Kubeflow will create a Kubernetes Pod to execute the pipeline component.

Verify Pod Creation

Monitor the pipeline execution from the Kubeflow dashboard or by checking the Kubernetes Pods.

Each pipeline component runs as an individual Kubernetes Pod.

View the Pipeline Logs

Open the component logs after the pipeline completes successfully.

Example output:

{  
  "parameterValues": {  
    "Output": "Hello, Pavan Kalyan!"  
  }  
}  

This confirms that the pipeline executed successfully and returned the expected output.

Real-World Example: Iris Classification Pipeline

The following example demonstrates a complete machine learning workflow using the Iris dataset. The pipeline loads the dataset, trains a machine learning model, evaluates its performance, and reports the model accuracy.

from typing import NamedTuple, List  
import kfp  
from kfp import dsl

@dsl.component(  
    base_image="python:3.8-slim",  
    packages_to_install=["pandas", "scikit-learn"]  
)  
def load_data() -> NamedTuple("Outputs", [("features", List[List[float]]), ("labels", List[int])]):  
    from sklearn.datasets import load_iris  
    iris = load_iris()  
    return (iris.data.tolist(), iris.target.tolist())

@dsl.component(  
    base_image="python:3.8-slim",  
    packages_to_install=["scikit-learn"]  
)  
def train_model(features: List[List[float]], labels: List[int]) -> NamedTuple("Output", [("accuracy", float)]):  
    from sklearn.ensemble import RandomForestClassifier  
    from sklearn.model_selection import train_test_split  
    from sklearn.metrics import accuracy_score

    X_train, X_test, y_train, y_test = train_test_split(features, labels)

    model = RandomForestClassifier()

    model.fit(X_train, y_train)

    acc = accuracy_score(y_test, model.predict(X_test))

    print(f"Accuracy: {acc}")

    return (acc,)

@dsl.pipeline(name="iris-ml-pipeline")  
def iris_pipeline():  
    data = load_data()  
    train_model(  
        features=data.outputs["features"],  
        labels=data.outputs["labels"]  
    )

if __name__ == "__main__":  
    kfp.compiler.Compiler().compile(  
        iris_pipeline,  
        "iris_pipeline.yaml"  
    )  

This pipeline performs the following tasks:

• Loads the Iris dataset.
• Splits the dataset into training and testing sets.
• Trains a Random Forest classifier.
• Evaluates the model using prediction accuracy.
• Returns the accuracy as the pipeline output.

Upload and Execute the Pipeline

Upload the generated iris_pipeline.yaml through the Kubeflow Pipelines dashboard.

After the upload completes:

1. Click Create Run.
2. Review the pipeline configuration.
3. Click Yes to start execution.

Monitor Pipeline Execution

The Kubeflow dashboard displays the execution graph, component status, logs, and output artifacts as the pipeline progresses.

Successful execution indicates that each component completed without errors and the trained model achieved the reported accuracy.

Why Run These Kubeflow Pipelines?

Each pipeline in this guide serves a different purpose and helps you understand a specific aspect of Kubeflow Pipelines, from basic workflow execution to real-world machine learning and production-ready environments.

Hello World Pipeline

The Hello World pipeline is the simplest Kubeflow Pipeline and is primarily used to verify that your Kubeflow environment is functioning correctly.

Purpose

• Validate the Kubeflow Pipelines installation.
• Understand the basic structure of a pipeline.
• Learn how components execute inside Kubernetes Pods.
• Verify pipeline compilation and execution.
• Familiarize yourself with the Kubeflow user interface.

This pipeline simply prints a greeting message and does not perform any machine learning tasks.

Iris Classification Pipeline

The Iris Classification Pipeline demonstrates a complete machine learning workflow using a real dataset.

Purpose

• Load a machine learning dataset.
• Train a machine learning model.
• Evaluate model performance.
• Execute multiple pipeline components.
• Understand data flow between pipeline stages.
• Learn how ML workflows are automated using Kubeflow.

This example introduces practical ML concepts while showcasing how Kubeflow orchestrates an end-to-end machine learning pipeline.

Custom Docker Image Pipeline

The Custom Docker Image Pipeline demonstrates how to execute Kubeflow components using a custom-built Docker image instead of the default runtime.

Purpose

• Use pre-installed machine learning libraries.
• Include custom operating system packages.
• Add organization-specific dependencies.
• Maintain consistent execution environments.
• Eliminate repeated package installation during pipeline execution.
• Build production-ready ML pipelines.

Custom images are particularly useful for complex ML projects that require specialized frameworks, GPU libraries, or additional system tools.

Comparison of the Three Pipeline Examples

Benefits for Machine Learning Engineers

Kubeflow Pipelines simplify the development, execution, and management of machine learning workflows.

Using the techniques covered in this guide, ML Engineers can:

• Automate end-to-end machine learning workflows.
• Execute reproducible experiments with consistent environments.
• Build reusable pipeline components.
• Integrate notebooks, training, evaluation, and deployment into a single workflow.
• Use custom Docker images for specialized ML frameworks and dependencies.
• Improve collaboration across data science and engineering teams.
• Reduce manual effort and minimize operational errors.

Benefits for Cloud Engineers

Kubeflow also provides valuable experience for Cloud Engineers responsible for building and managing cloud-native infrastructure.

This guide demonstrates how to:

• Deploy Kubernetes clusters using KIND.
• Install and configure Kubeflow Pipelines.
• Manage containerized machine learning workloads.
• Build and publish Docker images.
• Understand Kubernetes-based workflow orchestration.
• Support scalable ML infrastructure using Kubernetes.
• Prepare environments for GPU workloads, storage integration, and autoscaling.

These skills are directly applicable to production Kubernetes environments running on major cloud platforms such as Google Cloud, AWS, and Azure.

Conclusion

Kubeflow Pipelines provide a structured and scalable approach to automating machine learning workflows on Kubernetes. Instead of manually executing individual tasks, you define each step as part of a reusable pipeline, making ML processes more reliable, reproducible, and easier to maintain.

Throughout this guide, you learned how to:

• Understand the architecture and core components of Kubeflow.
• Build machine learning workflows using Kubeflow Pipelines.
• Create reusable pipeline components.
• Build and use custom Docker images.
• Deploy Kubeflow Pipelines on a KIND Kubernetes cluster.
• Execute sample pipelines ranging from a simple Hello World example to a real-world Iris Classification workflow.

Whether you're an ML Engineer, Cloud Engineer, DevOps Engineer, or Platform Engineer, Kubeflow Pipelines provide the foundation for building modern, cloud-native MLOps platforms.

If you're looking to build scalable, reproducible, and production-ready machine learning systems, learning Kubeflow Pipelines is an essential step toward mastering MLOps.