Ansible: Idempotent Playbooks

Ansible enables infrastructure as code. It has been a valuable tool for my infrastructure@home project, helping me to setup the programs Consul and Nomad, to provide DNS for nodes and services, and an Nginx endpoint. Although this infrastructure was completely replaced by Kubernetes, Ansible is a tool that stayed and provides continuous support for system maintenance tasks.

Having worked with Ansible for a longer time, my scripts now reside in one well-structured directory. From here, I can run individual tasks as well as running the complete site.yml to apply one consistent configuration state to all systems. It is also helpful to consider uninstallers that will remove unwanted configuration or software. And to achieve idempotency of your playbook, it is important to truly understand the Ansible modules that you are using. All of these is detailed in this article.

This article originally appeared at my blog.

Effective Directory Layout

All ansible code is contained within one single directory. At the time of writing this article, this directory looks like this:

├── ansible.cfg
├── host_vars
│   ├── raspi-3-1.yml
│   ├── ...
├── group_vars
├── roles
│   ├── consul
│   ├── docker-arch
│   ├── docker-arm
│   ├── nfs-client
│   ├── nfs-server
│   └── nomad
├── scripts
│   ├── consul
│   ├── nomad
│       ├── configs
│       ├── jobs
│   ├── system
│       ├── update_packages.yaml
│   ├── tutorial
│   └── uninstall
├── hosts
└── site.yml

The directory is structured according to these principles:

• Global config files in directory root: The ansible config files ansible.cfg and the inventory hosts.
• Global playbook: The playbook site.yml is an idempotent playbook that, when executed, configures all nodes with all the infrastructure systems that I have. Effectively it installs all the roles on all nodes and patches the nodes to the newest OS packages.
• Global vars: The group_vars directory contains global variables, especially the IP addresses for the Nomad, Consul and NFS servers. In the host_vars directory, I include a file for each node that determines its Nomad/Consul/NFS role as being master or agent.
• Separating roles and scripts: The roles directory contains playbooks that install infrastructure systems. These are the ansible roles I explained in earlier articles. In scripts are commands that are executed regularly, things like updating the nodes or restarting processes on the nodes. I also include the deployments of Nomad jobs, including config files for programs, inside the script directory.

Uninstaller

Ansible is concerned with providing repayable, consistent configuration for systems. But sometimes you need to undo these changes. If it’s just uninstalling software with the respective package management software, this is an easy task. But what about other configuration: Custom service files, configuration files, added configuration lines to central system files like /etc/mount or /etc/ssh/sshs_config?

There is no uninstaller in Ansible, but every change you make can be undone. Write this uninstaller yourself! Write it as soon as you are finished with the installation part, because its fresh on your mind and you can test it with your system.

I structure the playbooks into an install and uninstall block. These blocks are invoked by passing the uninstall=true parameter to the run.

Here is a shortened example for installing dnsmasq.

- block:
    - name: Install dnsmasq
      apt:
        name: dnsmasq
        state: present
    - name: Configure dnsmasq
      lineinfile:
        path: /etc/dnsmasq.d/10_consul
        create: true
        line: server=/consul/192.168.2.201#8600
        regexp: consul
        state: present
  when: uninstall is not defined
- block:
    - name: Uninstall
      apt:
        name: dnsmasq
        state: absent
    - file:
        path: /etc/dnsmasq.d
        state: absent
  when: uninstall is defined and uninstall

When this playbook is run as ansible-playbook ... -e "{uninstall: true}, the output is as follows:

ansible-playbook site.yml --limit=raspi-3-1 --tags dns -e "{uninstall: true}"

PLAY [Configure DNS] ***********************************************************************************************************************
TASK [dns : Install dnsmasq] ***************************************************************************************************************
skipping: [raspi-3-1]
TASK [dns : Configure dnsmasq] *************************************************************************************************************
skipping: [raspi-3-1]
TASK [dns : Restart dnsmasq] ***************************************************************************************************************
skipping: [raspi-3-1]
TASK [dns : Uninstall] *********************************************************************************************************************
changed: [raspi-3-1]
TASK [dns : file] **************************************************************************************************************************
changed: [raspi-3-1]

Idempotent Playbooks

When a playbook is executed to configure a system, the system should always have the same, well defined state. If a playbook consists of 10 steps, and the system deviates in step 4 from the desired state, then only this particular step should be applied.

By its nature, Ansible tasks will only change the system if there is something to do. Most Ansible modules provide this idempotency. But some modules can be used in a way that breaks this pattern. I cannot cover all modules, but want to highlight this point with one example: The module Line in File.

Let’s consider the setup of mounting an NFS volume. On the clients, you need to add an entry to etc/fstab for telling the system where the NFS server is. The first version of this change is this:

- name: Create fstab entry
  lineinfile:
    path: /etc/fstab
    line: '{{ nfs_dir_server_ip }}:{{ nfs_dir_mnt_path }}  {{ nfs_dir_mnt_path }} nfs defaults,soft,bg,noauto,rsize=32768,wsize=32768,noatime  0 0'
    state: present

The particular output is dependent on three variables. If one of the variables change, and the playbook is executed again, it will create a new entry into /etc/fstab. The mounts will not work!

You want only one entry for NFS shares in this file. And for this, you can use the additional regexp parameter. The second version of this playbook uses the regexp parameter to only and ever add one line to /etc/fstab. This simple change makes the playbook idempotent.

- name: Create fstab entry
  lineinfile:
    path: /etc/fstab
    line: '{{ nfs_dir_server_ip }}:{{ nfs_dir_mnt_path }}  {{ nfs_dir_mnt_path }} nfs defaults,soft,bg,noauto,rsize=32768,wsize=32768,noatime  0 0'
    regex: 'nfs defaults,soft,bg,noauto'
    state: present

Whenever you use a module, take care to use it idempotently.

Conclusion

During continuous use of Ansible, my collection of individual scripts evolved into one effective directory structure. Knowing exactly where to put which playbook, variables or configuration files is essential. Also, when writing a playbook, always add an uninstaller so you can undo changes to the system. Finally, be sure to write idempotent playbooks so that repetitive playbook execution always results in one well defined, consistent system state.