For anyone that wants to experiment with a highly available kubernetes cluster without shelling out a significant amount of money to a cloud provider, there’s no better way to get started than K3s. This small kubernetes distro is meant for resource-constrained devices, making it the perfect fit for a Pi. I was able to successfully run a K3s cluster on 3 x Raspberry Pi 4s with 4GB of RAM each.

There is another kubernetes distro I tried called (Microk8s) which is equally easy to setup and cluster. However, MicroK8s has greater resource requirements and recommend at least 4GB of RAM compared to K3s’ 1GB recommendation.

Setup

I started by installing Ubuntu Server 20.10 on each on each Pi. You can use the Raspberry Pi imager tool to flash a SD card/USB drive.

You will also need to enable cgroups which aren’t enabled by default. To enable this on Ubuntu, add the following snippet

cgroup_enable=memory cgroup_memory=1

to the end of the file /boot/firmware/cmdline.txt.

Optional:

I also added all my hosts into the /etc/hosts file. I refer to my hosts by hostname rather than IP when adding subsequent nodes. This is not necessary and you could use the node’s IP address instead.

Network Setup

I also recommend that you assign a static IP to each of your nodes to ensure they are always reachable at the same IP address on your network. I did this by configuring my router to assign a fixed IP based on the device MAC address.

Bootstrapping the first node

Run the following command on the first host to start a K3s node in server mode and to init a cluster.

curl -sfL https://get.k3s.io | sh -s - server  --cluster-init

K3s ships with a built in copy of kubectl. To use this version of kubectl without sudo, you can make the kubectl config readable by all users.

sudo chmod a+r /etc/rancher/k3s/k3s.yaml

You can now check the status of your cluster using kubectl.

ubuntu@pi0:~$ k3s kubectl cluster-info
Kubernetes master is running at https://127.0.0.1:6443
CoreDNS is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
Metrics-server is running at https://127.0.0.1:6443/api/v1/namespaces/kube-system/services/https:metrics-server:/proxy

Adding additional nodes

Adding more nodes is trivial. From each additional node, run the following command replacing pi0 with the hostname for your first node.

curl -sfL https://get.k3s.io | sh -s - server  --server https://pi0:6443
sudo chmod a+r /etc/rancher/k3s/k3s.yaml

And that’s it! If you now run get nodes, you should see all your nodes listed.

ubuntu@pi0:~$ k3s kubectl get nodes
NAME   STATUS   ROLES         AGE   VERSION
pi0    Ready    etcd,master   45h   v1.19.4+k3s1
pi1    Ready    etcd,master   45h   v1.19.4+k3s1
pi2    Ready    etcd,master   45h   v1.19.4+k3s1

Testing the cluster

To simulate a node going offline, you can reboot one of your node. The node that’s offline will be marked as NotReady.

ubuntu@pi0:~$ k3s kubectl get nodes
NAME   STATUS     ROLES         AGE   VERSION
pi0    Ready      etcd,master   46h   v1.19.4+k3s1
pi1    NotReady   etcd,master   46h   v1.19.4+k3s1
pi2    Ready      etcd,master   46h   v1.19.4+k3s1

After a while though when your node has rebooted, you should see all nodes listed as Ready once again.

Future Work

I’ll soon be posting about I use K3s’ local path provisioner for creating persistent volumes.

In the meanwhile, happy experimentation with your kubernetes cluster!