Setting Up a Multi-Node Kubernetes Cluster with Kind and Cilium CNI

Introduction

The purpose of this article is to demonstrate how to use a Container Network Interface (CNI) for networking in a multi-node Kubernetes cluster. Specifically, we will:

• Set up a two-node Kubernetes cluster using Kind (Kubernetes in Docker).
• Disable the default CNI to allow the installation of a custom CNI plugin.
• Install Cilium, one of the most popular CNI plugins.

This hands-on guide is intended for beginners who want to set up Kubernetes locally and explore how Container Network Interfaces work. If this article is well received, it may become part of a series covering Kubernetes, CNIs, and operators.

Understanding the Tools and Concepts

1. What is Cilium?

Definition

Cilium is an open-source networking and security solution for connecting and protecting container workloads.

It operates at the network layer to provide advanced networking capabilities such as:

• Load balancing
• Network policies
• Observability and monitoring

Use Cases

• Networking: Improves communication between containers with high-performance networking.
• Security: Enables fine-grained network policies to control which containers can communicate with one another.

Example

Imagine you have a fleet of delivery trucks (containers) that need to communicate to coordinate deliveries. Cilium acts as the central traffic management system that directs traffic efficiently while ensuring that only authorized trucks can communicate with each other, preventing security breaches.

2. What is CNI?

Definition

Container Network Interface (CNI) is a standard for configuring network interfaces for Linux containers.

It provides a specification and libraries for configuring network connectivity, making it easy to integrate different networking solutions with Kubernetes.

Use Cases

• Standardization: Provides a consistent way to assign networking resources to containers.
• Integration: Enables different networking plugins to work seamlessly with Kubernetes.

Example

Think of CNI as a universal electrical outlet in a building. Regardless of the appliance (network plugin), it can plug into the same outlet and work correctly.

3. What is Cilium CNI?

Definition

Cilium CNI is a Kubernetes CNI plugin powered by Cilium.

It leverages eBPF (extended Berkeley Packet Filter) technology to deliver efficient, scalable networking and advanced security capabilities.

Use Cases

• High-performance Kubernetes networking
• Fine-grained security policies
• Secure inter-container communication

Example

Continuing the office analogy, Cilium CNI is like an advanced security team equipped with state-of-the-art technology that manages and secures every communication inside the office.

4. What is Kind?

Definition

Kind (Kubernetes in Docker) is a tool that runs local Kubernetes clusters using Docker containers as cluster nodes.

It is primarily intended for development and testing.

Use Cases

• Local Kubernetes development
• Multi-node cluster testing
• Learning Kubernetes without cloud infrastructure

Example

Imagine planning a new office layout before making permanent changes. Kind lets you build and test a miniature version of your office in your living room before implementing it in the real office.

5. What is Helm?

Definition

Helm is the package manager for Kubernetes, similar to apt or yum for Linux.

It helps define, install, upgrade, and manage Kubernetes applications using Helm Charts.

Use Cases

• Simplifies Kubernetes deployments
• Supports application versioning
• Enables deployment rollbacks
• Manages application lifecycle

Example

Helm is like an app store for your Kubernetes cluster. Instead of manually installing software, you can deploy and manage applications using a few simple commands.

6. What is Docker?

Definition

Docker is a platform for developing, shipping, and running applications inside lightweight containers.

Containers package applications together with their dependencies, ensuring consistent behavior across environments.

Use Cases

• Application portability
• Environment consistency
• Process isolation
• Efficient resource utilization

Example

Think of Docker as a shipping container for software. Just as shipping containers transport goods safely across the world, Docker containers package applications so they run consistently on any system.

7. What is Kubernetes?

Definition

Kubernetes is an open-source platform that automates the deployment, scaling, and management of containerized applications.

It orchestrates clusters of machines to efficiently run and manage containerized workloads.

Use Cases

• Automated deployments
• Automatic scaling
• Self-healing workloads
• Load balancing
• Resource management

Example

Kubernetes is like a skilled office manager who ensures every department (container) has the resources it needs, scales teams when workloads increase, and keeps everything running smoothly with minimal manual intervention.

Prerequisites

Before you begin, ensure the following tools are installed on your EC2 instance:

• Docker
• Kind
• kubectl

Install Docker

Update the package index and install Docker:

sudo apt update  
sudo apt install -y docker.io  

Start and enable the Docker service:

sudo systemctl start docker  
sudo systemctl enable docker  

Add the current user to the Docker group to run Docker commands without sudo:

sudo usermod -aG docker ubuntu  

Note: Log out and log back in (or restart the instance) for the group membership changes to take effect.

Install Kind

Download and install Kind (Kubernetes in Docker):

curl -Lo ./kind https://kind.sigs.k8s.io/dl/v0.11.1/kind-linux-amd64

chmod +x ./kind

sudo mv ./kind /usr/local/bin/kind  

Install kubectl

Download the latest stable release of kubectl:

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl.sha256"  

Verify the downloaded binary:

echo "$(cat kubectl.sha256) kubectl" | sha256sum --check  

Install kubectl:

sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl

chmod +x kubectl

mkdir -p ~/.local/bin

mv ./kubectl ~/.local/bin/kubectl  

Verify the installation:

kubectl version --client  

Step 1: Create a Multi-Node Kubernetes Cluster Using Kind

The first step is to create a Kind configuration file that defines a multi-node Kubernetes cluster.

1.1 Create a Kind Configuration File

Create a file named kind-config.yaml with the following content:

kind: Cluster  
apiVersion: kind.x-k8s.io/v1alpha4

nodes:  
  - role: control-plane  
  - role: worker  
  - role: worker  

Note: The configuration above creates one control-plane node and two worker nodes (three nodes in total). If you intend to create a two-node cluster, remove one of the worker node entries.

1.2 Create the Cluster

Run the following command to create the cluster:

kind create cluster --config kind-config.yaml  

This command uses Docker to provision the Kubernetes cluster based on the configuration defined in kind-config.yaml.

1.3 Verify the Cluster

Verify that the cluster is running successfully:

kubectl cluster-info  

You should see information about the Kubernetes control plane and other cluster components.

Step 2: Disable the Default CNI Plugin

By default, Kind installs its own networking components. Before installing Cilium, remove the default networking configuration.

2.1 Remove kube-proxy

Delete the kube-proxy DaemonSet:

kubectl delete daemonset -n kube-system kube-proxy  

This removes the default Kubernetes networking proxy, allowing Cilium to replace it.

2.2 Remove Existing CNI Configuration

Delete any existing CNI configuration files from the node:

sudo rm /etc/cni/net.d/*  

This prevents conflicts between the default CNI and Cilium during installation.

Step 3: Install the Cilium CNI Plugin

Cilium provides high-performance networking, observability, and security for Kubernetes using eBPF.

3.1 Install Helm

If Helm is not already installed, install it using the following command:

curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash  

Verify the installation:

helm version  

3.2 Add the Cilium Helm Repository

Add the official Cilium Helm repository:

helm repo add cilium https://helm.cilium.io/

helm repo update  

3.3 Install Cilium

Install Cilium using Helm:

helm install cilium cilium/cilium \  
  --version 1.11.0 \  
  --namespace kube-system  

This deploys Cilium into the kube-system namespace and configures it as the cluster's CNI plugin.

3.4 Verify the Cilium Installation

Check that the Cilium pods are running successfully:

kubectl get pods -n kube-system -l k8s-app=cilium  

Note: The original command used -1, which appears to be a typo. The correct option is the lowercase letter -l (label selector).

If the installation is successful, you should see the Cilium pods in the Running state.

FAQs

1. What are Network Policies in Kubernetes?

Network Policies define rules that control how traffic flows between pods in a Kubernetes cluster. They help improve security by restricting communication, implementing access controls, and enforcing network segmentation.

2. How can I verify that Cilium is working correctly?

You can verify a successful Cilium installation by:

• Checking that the Cilium pods are in the Running state:

  kubectl get pods -n kube-system -l k8s-app=cilium  
  

• Ensuring there are no errors in the pod logs.
• Deploying sample applications and validating connectivity between pods.
• Testing Network Policies to confirm that traffic is allowed or denied as expected.

3. Can I use Cilium alongside another CNI plugin?

Cilium is designed to function as the primary CNI plugin in a Kubernetes cluster. Running multiple CNI plugins simultaneously is generally not recommended unless using a supported migration or chaining scenario documented by the Cilium project.

Conclusion

• Created a multi-node Kubernetes cluster using Kind.
• Disabled the default networking components.
• Installed Cilium as the cluster's CNI plugin.
• Verified that the Cilium installation was successful.

This setup provides a solid foundation for learning Kubernetes networking and experimenting with advanced CNI capabilities such as network policies, observability, and security. As you become more familiar with Cilium, you can further explore features like eBPF-powered networking, Hubble observability, ingress, and service mesh capabilities.

If you found this guide useful, stay tuned for future articles covering Kubernetes, CNIs, and Kubernetes operators in greater depth.