Lighthouse Scanner: How To Setup a Local Kubernetes Staging Environment

With my Lighthouse-as-a-Service website scanner you can quickly check a webpage about its performance, SEO and best practices. You can use the scanner here: https://lighthouse.admantium.com/.

From an application perspective, my lighthouse scanner was a monolith until recently: One docker container that delivers the webpage, has an API for accepting scan requests, and executes the scans. Testing was simple: Just start the Docker container. However, I redesigned the monolith to separate microservices, and now I have three Docker containers plus a database that communicate with each other.

In my production environment, I’m using Kubernetes. Containers are created with a Deployment, persistent volumes are available, and further application configuration in the form of ConfigMaps and Secrets. I want to have an easy way of testing this setup in a stage environment. I want the complexity of microservice communication in my staging environment. And ultimately, I want to change as little as possible in my build commands and configurations.

With some planning ahead, you can achieve the very same thing. In this article, I will highlight the most important aspects: Setup local Kubernetes, configuration management, container communication and multiarch builds.

This article originally appeared at my blog.

Local Kubernetes Staging Environment

For the staging environment I do not want to use my MacBook, but also an environment of multiple nodes. And for this, I can happily use my Raspberry Pi Stack. It turns out the K3S can be installed as easily on Raspberry pi. To convince you: For setting up a two-node cluster, you just need to execute the following two commands. That is all.

k3sup install --ip $SERVER_IP --user $USERk3sup join --ip $NODE_IP --server-ip $SERVER_IP --user $USERNAME

Now I have a production and staging environment with the same technology. To switch between the environments, I simply change the used .kubeconfig file to .kubconfig-prod or .kubeconfig-stage. And with this, most other artifacts can be reused without much changes.

Unified Configuration Management

  • Docker image: Copy config files into the container or define fixed environment variables
  • Docker container: Mount config files into the container, define runtime environment variables
  • Kubernetes Deployments: Mount config files or environment variables into the containers

Since I’m using Kubernetes as the execution environment for production and staging, I want to provide all configuration options with Kubernetes.

One example: The configuration for Redis moved from the Docker container to a Kubernetes ConfigMap and volume mount.

apiVersion: v1
kind: ConfigMap
metadata:
name: redis-config
data:
redis.conf: |
save 60 1
appendonly yes
...
apiVersion: apps/v1
kind: Deployment
metadata:
name: redis
spec:
replicas: 1
...
spec:
containers:
- name: redis
image: docker.admantium.com/redis-multi:latest
...
volumeMounts:
- name: redis-config
mountPath: /usr/local/etc/redis/
...

ConfigMaps and Secrets are defined declaratively in yaml files. They do not differ in their values, and I just need to apply them to the production or staging environment.

Simplifying Container Communication

In Kubernetes, this is delightfully simple. A Deployment is exposed with a Service. The service name becomes the domain name. Here is an example for Redis.

apiVersion: v1
kind: Service
metadata:
name: redis
spec:
ports:
- protocol: TCP
port: 7139
targetPort: 7139
selector:
app: redis

This service definition defines the domain name redis which is available from each other Pod running in the cluster. Delightfully simple.

Since all containers are in the same namespace, I can simply reuse the very same service definitions and therefore the domain names in production and staging environment.

Docker Multiarch Build

The full steps and benefits of using multiarch builds are detailed in my article Building Images for Multiple Architectures. In a nutshell:

  • Install Docker Community Edition version 2.0.4.0 or higher
  • Enable experimental features
  • Create a multiarch builder object
  • Use docker buildx build --platform

Multiarch Docker images provide the very same execution environment for your application, a crucial aspect for testing. The only change needed for me was to use the new buildx command.

Conclusion

IT Project Manager & Developer