Lighthouse Scanner: How To Setup a Local Kubernetes Staging Environment
With my Lighthouse-as-a-Service website scanner you can quickly check a webpage about its performance, SEO and best practices. You can use the scanner here: https://lighthouse.admantium.com/.
From an application perspective, my lighthouse scanner was a monolith until recently: One docker container that delivers the webpage, has an API for accepting scan requests, and executes the scans. Testing was simple: Just start the Docker container. However, I redesigned the monolith to separate microservices, and now I have three Docker containers plus a database that communicate with each other.
In my production environment, I’m using Kubernetes. Containers are created with a Deployment
, persistent volumes are available, and further application configuration in the form of ConfigMap
s and Secrets
. I want to have an easy way of testing this setup in a stage environment. I want the complexity of microservice communication in my staging environment. And ultimately, I want to change as little as possible in my build commands and configurations.
With some planning ahead, you can achieve the very same thing. In this article, I will highlight the most important aspects: Setup local Kubernetes, configuration management, container communication and multiarch builds.
This article originally appeared at my blog.
Local Kubernetes Staging Environment
My production environment uses K3S, a lightweight Kubernetes distribution. Its setup consists of exactly one line per host. On top of that, I created a private docker registry, an Nginx Ingress, and the cert manager for TLS encryption.
For the staging environment I do not want to use my MacBook, but also an environment of multiple nodes. And for this, I can happily use my Raspberry Pi Stack. It turns out the K3S can be installed as easily on Raspberry pi. To convince you: For setting up a two-node cluster, you just need to execute the following two commands. That is all.
k3sup install --ip $SERVER_IP --user $USERk3sup join --ip $NODE_IP --server-ip $SERVER_IP --user $USERNAME
Now I have a production and staging environment with the same technology. To switch between the environments, I simply change the used .kubeconfig
file to .kubconfig-prod
or .kubeconfig-stage
. And with this, most other artifacts can be reused without much changes.
Unified Configuration Management
In Kubernetes, applications configuration can happen in several places:
- Docker image: Copy config files into the container or define fixed environment variables
- Docker container: Mount config files into the container, define runtime environment variables
- Kubernetes
Deployments
: Mount config files or environment variables into the containers
Since I’m using Kubernetes as the execution environment for production and staging, I want to provide all configuration options with Kubernetes.
One example: The configuration for Redis moved from the Docker container to a Kubernetes ConfigMap
and volume mount.
apiVersion: v1
kind: ConfigMap
metadata:
name: redis-config
data:
redis.conf: |
save 60 1
appendonly yes
...apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
spec:
replicas: 1
...
spec:
containers:
- name: redis
image: docker.admantium.com/redis-multi:latest
...
volumeMounts:
- name: redis-config
mountPath: /usr/local/etc/redis/
...
ConfigMaps
and Secrets
are defined declaratively in yaml files. They do not differ in their values, and I just need to apply them to the production or staging environment.
Simplifying Container Communication
A short detour. I used Hashicorp Consul and Nomad to manage container applications before switching to Kubernetes. In order to achieve container-communication with domain names like redis.local
, I needed to setup a local DNS server on each node, configure this server to connect to the Consul service discovery, and provide an Nginx reverse proxy configuration to get TLS encryption. Read about these details in Service Discovery with Consul and Traefik and Nginx as a Reverse Proxy.
In Kubernetes, this is delightfully simple. A Deployment
is exposed with a Service
. The service name becomes the domain name. Here is an example for Redis.
apiVersion: v1
kind: Service
metadata:
name: redis
spec:
ports:
- protocol: TCP
port: 7139
targetPort: 7139
selector:
app: redis
This service definition defines the domain name redis
which is available from each other Pod running in the cluster. Delightfully simple.
Since all containers are in the same namespace, I can simply reuse the very same service definitions and therefore the domain names in production and staging environment.
Docker Multiarch Build
Using Docker Multiarch Builds is an optional step. In my setup, the production environment runs on linux/amd64 and the staging is linux/arm/v7. With the goal of changing as few things as possible, building Docker images that run on multiple platforms is a necessity. This saves the trouble of using different machines for builds, using different Dockerfiles, naming/taging the images and use these different images in your deployment.
The full steps and benefits of using multiarch builds are detailed in my article Building Images for Multiple Architectures. In a nutshell:
- Install Docker Community Edition version 2.0.4.0 or higher
- Enable experimental features
- Create a multiarch builder object
- Use
docker buildx build --platform
Multiarch Docker images provide the very same execution environment for your application, a crucial aspect for testing. The only change needed for me was to use the new buildx
command.
Conclusion
Testing applications is important. In complex microservice setup, you should recreate a staging environment that is as close as possible to your production environment. In the context of Kubernetes, this article outlined the important aspects: Use the same Kubernetes distribution. Move all application configuration from Docker to Kubernetes ConfigMaps
and Secrets
. Use consistent Service
definition for simplifying container communication. And, optionally, use Docker multiarch builds to support different hardware execution environments. Overall, this approach requires time spend in environment setup and tooling modification, and rewards you with a near 100% reuse of build commands, application configuration and deployment steps.