Persisting Data with Nomad

Nomad Docker Bind Mounts

  • The Nomad agent config needs a host_volume config
  • The job group needs a volume declaration
  • The tasks needs a volume_mount declaration

nomad.conf.hcl

client {
host_volume "prometheus_vol" {
path = "/mnt/nfs/nomad/volumes/prometheus"
read_only = false
}
}

monitoring.job.hcl

group "monitoring" {
volume "prometheus_grp" {
type = "host"
source = "prometheus_vol"
read_only = false
}
task "grafana" {
volume_mount {
volume = "prometheus_vol"
destination = "/prometheus"
readonly = false
}
...
}

Nomad Docker Volume Mounts

>> docker volume lsDRIVER              VOLUME NAME
local 1e9b98423c15f1aa833b131ab9f6c72b0433a212a89b5b75a30424ba917f09d9
local prometheus_grp
>> docker volume inspect prometheus_grp
[
{
"CreatedAt": "2020-03-14T11:54:04Z",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/prometheus_grp/_data",
"Name": "prometheus_grp",
"Options": null,
"Scope": "local"
}
]

monitoring.job.hcl

task "grafana" {
driver = "docker"
config {
mounts = [
{
type = "volume"
target = "/prometheus"
source = "prometheus"
}
]
}
...
}

Day 1 — Trial & Error

Step 1: Symlink Docker Volumes to NFS

>> ls -la /var/lib/docker/volumes/prometheus_grp/_data/total 16
drwxr-xr-x 3 nobody nogroup 4096 Mar 14 11:54 .
drwxr-xr-x 3 root root 4096 Mar 14 11:54 ..
-rw-r--r-- 1 nobody nogroup 0 Mar 14 11:54 lock
-rw-r--r-- 1 nobody nogroup 20001 Mar 14 12:13 queries.active
drwxr-xr-x 2 nobody nogroup 4096 Mar 14 12:13 wal

Second Try: Docker NFS Volumes

docker volume create --driver local \
--opt type=nfs \
--opt o=addr=192.168.2.203,rw \
--opt device=:/mnt/nfs/volumes/test/ nfs-test
docker run -v /mnt/nfs/volumes/test/:/nfs-test busybox touch /nfs-test/hellodocker: Error response from daemon: error while mounting volume '/var/lib/docker/volumes/nfs-test/_data': failed to mount local volume: mount /mnt/nfs/volumes/test:/var/lib/docker/volumes/nfs-test/_data, data: addr=192.168.2.203: invalid argument

Third Try: Docker Bind Volumes

Day 2 — Systematic Trial & Success

Figuring Out What’s Wrong

docker volume create
--driver local
--opt type=nfs
--opt "o=nfsvers=4,addr=192.168.2.203,nolock,soft,rw"
--opt device=:/mnt/nfs/volumes/test/ nfs-test
docker run -v /mnt/nfs/volumes/test/:/nfs-test busybox touch /nfs-test/hello

Nomad Named Volume Declaration

task "grafana" {
driver = "docker"
config {
mounts = [
{
type = "volume"
target = "/prometheus"
source = "prometheus"
}
]
...
}
}
failed to create container: API error (500): failed to mount local volume: mount :/mnt/nfs/docker/prometheus:/var/lib/docker/volumes/prometheus/_data, data: nfsvers=4,addr=192.168.2.203,nolock,soft: no such file or directory

Nomad Bind Mount Declaration

task "grafana" {
driver = "docker"
config {
mounts = [
{
type = "bind"
target = "/prometheus"
source = "/mnt/nfs/docker/prometheus"
readonly = false
bind_options {
propagation = "rshared"
}
}
]
...
}
}
>> docker volume inspect prometheus
[
{
"CreatedAt": "2020-03-18T19:15:57Z",
"Driver": "local",
"Labels": {},
"Mountpoint": "/var/lib/docker/volumes/prometheus/_data",
"Name": "prometheus",
"Options": {
"device": ":/mnt/nfs/docker/prometheus",
"o": "nfsvers=4,addr=192.168.2.203,nolock,soft,rw",
"type": "nfs"
},
"Scope": "local"
}
]
>> docker volume inspect prometheus[
{
"CreatedAt": "2020-03-18T19:17:30Z",
"Driver": "local",
"Labels": null,
"Mountpoint": "/var/lib/docker/volumes/prometheus/_data",
"Name": "prometheus",
"Options": null,
"Scope": "local"
}
]

Nomad Volume Declaration

task "grafana" {
driver = "docker"
config {
image = "prom/prometheus:v2.16.0"
volumes = [
"local/prometheus.yml:/etc/prometheus/prometheus.yml",
"/mnt/nfs/docker/prometheus:/prometheus"
]
...
}
}

Solution

  • Setup nfs share on nodes
  • Use a config.volumes declaration with direct mapping between host source path and container destination path
volumes = [
"/path/to/nfs/share:/path/to/docker/destination"
]

Conclusion

Footnotes

  1. As I learned later, that’s not correct: It will overwrite the docker volume declaration if it is anything else then a normal volume! If you defined an NFS volume with the same name, it will get overwritten.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store