Application Hosting: Bare Metal, Nomad or Kubernetes?

Application hosting is a complex choice. Do you just install the application and its dependencies on a server, and start the process with the native runtime binary? Are you Dockerizing your application and run a container? Are you using a scheduler like Nomad or Kubernetes?

I was faced with this decision for my lighthouse as a service application. This article explains and discusses the pro and cons of application hosting options. While my choice reflects my application requirements, I’m sure you can learn about the difference options and apply this knowledge when you make a similar decision.

More about lighthouse: This service scans webpages and reports how good they score on terms of SEO, performance and best practices. You can use the scanner here: https://lighthouse.admantium.com/.

This article originally appeared at my blog.

Application Deployment Options

Deployment comes down to a simple core requirement: Make the frontend reachable with a domain name, and make the backend accessible for the frontend.

This sounds easy — and then you think about aspects like TLS encryption, service discovery and communication, fetching log files, fault tolerance, auto-scaling, build pipeline … the list goes on.

So, what are the options? I will discuss three types:

• Bare metal: Nginx with static HTML to deliver the frontend, Docker containers for the backend. Installed with Ansible on each node
• Consul & Nomad: deployment and management of containers with Nomad, service discovery with Consul, load balancing with Nginx
• Kubernetes: deployment, management, service discovery and load balancing with Kubernetes components

Bare Metal

Required Steps

Frontend

• Install Nginx

Backend

• Install Docker
• Build and deploy backend as Docker container
• Configure Nginx upstream servers for backend
• Configure load balancing for backend

Deployment

• TLS: Generate TLS certificate, configure Nginx
• Frontend: Configure Nginx to deliver static HTML
• Backend: Build Docker containers, push and deploy on nodes
• Service Discovery: Configure Nginx upstream server group with static IP and ports
• Load Balancing: Configure Nginx upstream server

The installation and configuration steps could be automated with Ansible playbooks. Ansible roles for installing Nginx and Docker are available. A custom task would build and push the static HTML to the server. Build and deployment of the backend Docker container can be done with a shell script.

Features

• ✅ Automatic Installation
• ❓ Automatic Deployment
• ✅ TLS Encryption
• ✅ Service Discovery
• ❌ Health Check
• ✅ Load Balancing
• ❌ Logging/Monitoring
• ❌ Auto-Scaling
• ❓ Self-Healing

Consul & Nomad

Required Steps

Cluster Management

• Install Consul on each node
• Install Nomad on each node
• Encrypt Consul & Nomad traffic

Frontend

• Install Docker on each node
• Dockerize Frontend app

Deployment

• TLS: Generate TLS certificate, configure Nginx
• Frontend: Define deployment job with Nomad
• Backend: Define deployment job with Nomad
• Service Discovery: Add service definition to Nomad job, configure nodes to use Consul DNS server
• Health Check: Add service definition to Nomad job
• Load Balancing: Configure Nginx upstream server to use DNS names for the services provided by Consul

Installation steps for the cluster management software would be covered by Ansible playbooks. The Encryption of Consul & Nomad is a semi-automatic process, in which I need to perform rolling updates on the nodes. Installing Docker would be done with Ansible. and Dockerizing the Frontend app, and getting the TLS certificate for Nginx, is a manual process. The workload definition combines deployment, service discovery and health checks. For load balancing, you also need to setup the local nodes to use the Consul DNS server.

Features

• ✅ Automatic Installation
• ✅ Automatic Deployment
• ✅ TLS Encryption
• ✅ Service Discovery
• ✅ Health Check
• ✅ Load Balancing
• ✅ Logging/Monitoring
• ❌ Auto-Scaling
• ✅ Self-Healing

Kubernetes

Required Steps

Cluster Management

• Install Kubernetes on each node

Frontend

• Dockerize Frontend app

Deployment

• TLS: Install cert manager, define ClusterIssuer resource
• Frontend: Define Service and Deployment resource
• Backend: Define Service and Deployment resource
• Service Discovery: Automatic
• Health Check: Add checks to Deployment resource
• Load Balancing: Automatic

Installation would be covered by choosing a Kubernetes distribution1. Dockerizing the frontend app, and installing cert manager is a manual process. Then all the remaining work is to define the Kubernetes resource definitions.

Features

• ✅ Automatic Installation
• ✅ Automatic Deployment
• ✅ TLS Encryption
• ✅ Service Discovery
• ✅ Health Check
• ✅ Load Balancing
• ✅ Logging/Monitoring
• ✅ Auto-Scaling
• ✅ Self-Healing

Choice

To make a choice, I considered various aspects: Time to get it running, number of manual tasks, number of features.

At first, I wanted to have the fasted time to market as possible. For me, this would be the bare metal installation. I have Ansible roles for Nginx and Docker installation ready. Pushing HTML and Dockerfiles to the nodes, and even starting the Docker container, can be done with Ansible too. The application would work, but as shown, a lot of features would be missing. Yes, you can define a docker health checks and autostart. Yes, you can setup health monitoring and log scraping. But these involve manual tasks again, and you would have more moving pieces that you need to manage and get experienced with.

Deploying Lighthouse needs more reliability and out-of-the-box features. Is it Nomad & Consul, or Kubernetes?

I have collected experience with Consul & Nomad for my infrastructure@home project. The steps to create a cluster are almost automated with Ansible playbooks. Overall, it works. Yet I had some challenges: Setting up storage volumes, or providing persistent service discovery with DNS on the hosts and between Consul nodes. Also, configuring Traefic or Nginx for ingress traffic was not easy. And there are also unsolved problems: Load-balancing with Nginx between multiple Consul services for example, because DNS resolution for upstream servers is not possible with the open source version of Nginx2.

With Kubernetes, I tried it initially with Docker for Desktop in 2018. Also, at work most applications are hosted with Kubernetes, but we use a given set of Helmcharts for deployment and do not administrate the cluster ourselves. However, a growing interest and steady accumulation of knowledge about the Kubernetes made me curious. The one thing that got me really interested is the K3S with which you get a Kubernetes cluster by running exactly one command on each node! Consider this simplicity against the Consul & Nomad installation. Also, all following tasks will be done with writing resource definitions for Kubernetes. Staying within one abstraction level is a benefit I cannot emphasize enough.

Deployment with K3S

So, in the end I decided to use Kubernetes with K3S. In only 7 days, including improvements of frontend and backend code, Lighthouse is hosted with Kubernetes. I saw how so many details including TLS, DNS, IPs and Ports are automated and abstracted. Yes, the learning curve is steep, and it takes time to absorb how Kubernetes works. F or me, it seems that all the troubles I had before made me knowledgeable to flatten this curve. I’m using Kubernetes, and don’t want to use another option.

Here is a list of these moments to share the enthusiasm with you:

• When the cluster runs after you executed a one liner installation script per node
• Deploy a docker container, expose the service, and access it immediately
• Deploying, scaling, deleting containers with a powerful CLI
• Realize that the ingress is Nginx, and that the configuration options you know are all there, represented as annotations
• Deploy a TLS encrypted private Docker registry
• Using cert-manager to automatically create lets-encrypt certificates for your live services
• The power to scale from 1 to 100 containers with a simple kubectl command

I have just very readable, very declarative resource definitions that provide all of the required features out-of-the box. Take a look at this Deployment declaration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: lighthouse
spec:
  replicas: 6
  selector:
    matchLabels:
      app: lighthouse
  template:
    metadata:
      labels:
        app: lighthouse
    spec:
      containers:
        - name: lighthouse
          image: docker.admantium.com/lighthouse:0.1.4
      imagePullSecrets:
        - name: registry-secret

Conclusion

Kubernetes is the cluster management software to host the Lighthouse scanner. In only 7 days I set up the cluster on three nodes, created a private Docker registry, and deployed multiple containers running Lighthouse. It’s all there: TLS encryption, automatic health-check, service-discovery, load balancing and auto healing. No additional software on the nodes, no additional config for IP, ports and DNS is necessary. It just works. I highly recommend to make the switch to Kubernetes too.

Footnotes

1. Kubernetes can be installed with various distributions, ranging from enterprise level like OpenShift, Rancher or Canonical, to medium sized applications with MicroK8S and K3S, to local development Docker Desktop or Kubeadm. ↩
2. There is a trick to define Nginx config files with Consul Templates. Template files will listen on Consul DNS changes, rewrite the files, then reload Nginx. See official documentation. ↩