Kubernetes with kops

The final Kubernetes distribution to look at is called kops, a CLI tool for configuring, setup and maintaining a Kubernetes cluster inside cloud environments. At the time of writing, only amazon AWS is fully supported, and this will be the focus of this article. With kops, you ultimately define all configuration options in a detailed YAML file, giving you a single source for your complete cluster. This article explains everything you need to know for getting started with kops.

This article originally appeared at my blog admantium.com.

Distribution Overview

With kops, a default configured Kubernetes cluster will have these components:

• Storage: etcd
• CRI: containerd
• CNI: kubenet

Kops aims to get cloud agnostic. At the time of writing, the fully supported provider is AWS. Other supported cloud providers in beta status are GCE, OpenStack, Digital Ocean, and alpha support for Azure, Hetzner, and Spot Ocean. The following instructions represent only the AWS setup and installation.

Installation Architectures

As the other distributions too, kops recommends using a multi controller, multi worker nodes architecture. But you can change the configuration to support other architectures as well.

• Single Node Cluster: Only configure a single server which hosts the control plane and can schedule workloads
• Single controller, multi worker: one single node is designated as the controller, and then additional worker nodes are joined that will host the workloads
• Multi controller, multi worker: The recommended architecture. Multiple controller nodes enable redundancy and fault tolerance, and multiple worker nodes can scale to host the various workloads.

For a high availability setup, following considerations need to be made:

• expose all nodes with a public topology, or use a private topology with a single load-balanced DNS access
• launch multiple controller nodes in different availability zones of AWS
• stack etcd instances on each controller node, or setup an external etcd cluster

For a production ready cluster, it is advised to not use the default CNI kubenet, because of several shortcomings, such as only supporting up to 50 nodes. See the documentation about kubenet support for more details.

Installation Process

To install a Kubernetes cluster with kops, you need a kops controller, a special machine on which the required AWS and kops binaries are installed. From this machine, you will provision the Kubernetes controller and worker nodes. Follow these steps:

AWS Configuration

• On the kops controller, install the AWS CLI
• Define an AWS IAM user with the correct access permissions
• Configure DNS-based access to your cluster by defining an AWS route 53 definition with a DNS name from Amazon or another external domain registration
• Define an AWS S3 bucket to store the kops state (kops supports multi-cluster setup/maintenance, and uses an S3 bucket to store the different configuration assets)

Cluster Configuration & Initialization

• Define in which AWS region the cluster is deployed, and determine the availability zones by running aws ec2 describe-availability-zones --region <region>
• Create a new cluster by running this command

kops create cluster \
    --name=${NAME} \
    --cloud=aws \
    --zones=eu-central-1a \
    --discovery-store=s3://my-k8s-cluster/${NAME}/discovery

• Configure the cluster by running kops edit cluster --name ${NAME}
• Finally create the cluster with kops update cluster --name ${NAME} --yes

kubectl configuration

• On the kops controller, use the kubeconfig file at location ~/.kube/config to access the cluster

Upgrade Process

Upgrading a Kubernetes installation with kops encompasses the following steps as outlined in the documentation:

Update the kops binary

• Install a new version of the kops binary to match the targeted Kubernetes version
• Edit the kops configuration file
• Run kops edit cluster $NAME
• Set the desired kubernetesVersion

Upgrade the cluster

• Run kops update cluster $NAME to see the changes, then run the same command again with the --yes flag
• Run kops rolling-update cluster $NAME to see the changes, then run the same command again with the --yes flag

In addition to upgrading the Kubernetes binaries, you can also upgrade the CNI without further considerations. Changing the CRI instead enforces installation steps similar to the ones for initial cluster setup.

Customization

Kubernetes installations with Kops can be configured to a great amount with a central YAML file. This file contains — in addition to the Kubernetes core components — additional aspects such as auto managing security updates for the nodes, using OpenID connect tokens, enable audit logging, or CPU management policies. See the complete specification.

The currently supported Kubernetes components are the following:

Control Plane Storage

• etcd

Container Runtime

• Containerd
• Docker (up to kops version v1.16)

Container Networking Interface

• AWS VPC
• Calico
• Canal
• Cilium
• Flannel
• Kopeio
• Kube Router
• Kubenet

There is no default Ingress controller or storage class defined.

Conclusion

The kops CLI is a powerful tool to setup a Kubernetes cluster running on AWS. For the initial preparation, you need a dedicated kops controller machine on which the AWS SDK is installed. Additionally, you need to setup an IAM user, a Rout53 DNS config, and an S3 bucket. This is used by the kops CLI to setup the cluster controller nodes and worker nodes. Almost all configuration items are present in one concise YAML file. Using declarative commands, the cluster is created and updated according to this configuration. Finally, updating the Kubernetes cluster is fully automated with kops.