#! / Hash / Bang / Wallop

Test-Driven Ansible with Molecule

Summary

How to TDD Ansible with Molecule, Docker, and Testinfra.

Introduction

As a Sys Admin type who likes to automate things I've learnt several configuration management tools over the years. The first one I learnt was Chef, in the Chef ecosystem they very much advocate that all cookbooks should have tests. In the Chef world testing is encouraged so much they even have canonical tools to help such as Test Kitchen You can actually use Test Kitchen to test other configuration management tools such as Ansible, Salt, Puppet etc. and InSpec. Whenever I was writing Chef I would use these tools to follow a test-driven development (TDD) just like I'd heard software developers banging on about when writing apps. I immediately fell in love.

I then discovered Ansible which I was attracted to for its simplicity in comparison to writing Ruby code with Chef. I immediately re-wrote all of the Chef stuff we had at work and persuaded the team to make the switch. I still love Chef, just Ansible was a lot quicker to understand which was going to mean much easier on-boarding of new team members

After writing a lot of Ansible I couldn't help think something was missing, it was the TDD workflow I'd fallen in love with when writing Chef. The Ansible community don't believe testing playbook is necessary on the same scale as it is with Chef as the execution of a playbook is strictly ordered and you are just writing YAML files rather than a full programming language. You'd almost be testing Ansible itself which already has a substantial amount of testing. Nevertheless I still missed the workflow and having tests gave me the confidence to refactor things or try new features out when learning. I missed having a 'clean' environment a command away to try out what I'd just written. Furthermore, some of the simplicity of Ansible starts to get lost once a million values have been abstracted out into variables or you start introducing logic to accommodate various operating systems, once this happens testing can't hurt.

I've since discovered Molecule which allows me to write Ansible in a very similar way to how I used to write Chef. I write tests first in Testinfra which is the Python equivalent to the Serverspec Molecule can use Serverspec for testing but it would require the installation of Ruby. Ansible itself is written in Python so let's stick with Python tools. tests I'd got used to writing for Chef. I had my workflow back and I have increased confidence in the Ansible code I write. In addition to testing Molecule ensures you follow best practices by running your code through ansible-lint and performs an idempotence check.

In this post I hope to show how you can use Molecule to TDD a simple Ansible role which installs Ruby from source. I will show how you can test with Testinfra and also include an example of testing within the Ansible itself which is often advocated. The final project can be found on Github.

Initial Setup

Although I'll show all the code step by step it is assumed that you already have some Ansible knowledge. You should also have a working Python 2.x I had issues with Python 3 when I tried, YMMV. setup along with a Docker install.

First off we will setup our project by setting up a virtual environment to store all the necessary Python dependencies and then we'll bootsrap Molecule itself. If you do not already have Virtualenv installed get it with pip install -U virtualenv, you may or may not need sudo privileges depending on your Python setup.

$ mkdir -p ~/ansible/ruby

$ cd ~/ansible/ruby

$ virtualenv venv

$ source venv/bin/activate

(venv) $ pip install molecule docker

As part of the Molecule install it will have installed Testinfra as a dependency, however at the time of writing, the current version (1.25.0) has a slightly outdated version. We want to upgrade to the latest Testinfra so that we can use the newer style fixtures that are available from 1.6.0 onwards when writing our tests. Molecule may indeed already have this by the time of reading as my humble contribution to upgrade it has been merged and the upcoming 2.0.0 also includes the latest version. In the meantime we can ensure we have the latest version that will be compatible with the tests I present like so:

(venv) $ pip install -U testinfra

Verify you have Testinfra 1.6.0 or above with:

(venv) $ pip list | grep testinfra

We now have all our project's dependencies isolated in a virtual environment. However, it is also good practice to create a requirements.txt file so that collaborators may also install the exact same dependencies. A collaborator can use the requirements file with: $ pip install -r requirements.txt.

(venv) $ pip freeze > requirements.txt

We are now ready to initialise Molecule, this will create some boilerplate files including a dummy test which we can use to make sure everything is setup correctly.

(venv) $ molecule init --driver docker

$ ls -lah
total 136K
drwxr-xr-x 5 rosstimson rosstimson   8 Jun  4 16:18 ./
drwxr-xr-x 3 rosstimson rosstimson   3 Jun  4 16:11 ../
drwxr-xr-x 2 rosstimson rosstimson   6 Jun  4 16:19 .molecule/
-rw-r--r-- 1 rosstimson rosstimson 208 Jun  4 16:18 molecule.yml
-rw-r--r-- 1 rosstimson rosstimson  43 Jun  4 16:18 playbook.yml
-rw-r--r-- 1 rosstimson rosstimson 811 Jun  4 16:17 requirements.txt
drwxr-xr-x 2 rosstimson rosstimson   4 Jun  4 16:18 tests/
drwxr-xr-x 8 rosstimson rosstimson   9 Jun  4 16:16 venv/

Molecule kindly lints all our Python code to make sure it is following best practices however there is a lot of third-party Python code in the virtualenv directory which we don't want to be included in this linting process as it will cause errors. Edit the molecule.yml file to ignore the venv directory - I have also removed the ansible_groups section as we are not goning to need it. You'll also notice I have switched to using the Amazon Linux image since this is the OS I primarily work with.

---
molecule:
  ignore_paths:
    - venv

dependency:
  name: galaxy

driver:
  name: docker

docker:
  containers:
    - name: ruby
      image: amazonlinux
      image_version: latest

verifier:
  name: testinfra

All being well we should now be able to successfully do a Molecule run, try it out:

(venv) $ molecule test

You should see a Docker container being created and prepared, followed by a short Ansible run. Now the container is in a converged state Molecule proceeds to run various tests such as an idempotence check, linting of both Ansible and Python, and lastly Testinfra.

Troubleshooting

If you are unable to successfully run molecule test a few common issues I've encountered are:

  • You are running the command from the wrong directory, this will result in the error: ERROR: Unable to find molecule.yml. Exiting.. Make sure you are in the same directory as the molecule.yml file.
  • Molecule has also been installed on the system globally. Depending on how your $PATH is setup this might mean that when you run the molecule command it is using that of the system install rather than within the virtualenv, this can cause all sorts of weird errors. Make sure that it is the virtualenv install that you are using by checking $ which molecule, this should show something like YOURHOME/ansible/ruby/venv/bin/molecule. A similar issue might occur if you forget to activate the virtualenv with $ source venv/bin/activate.
  • Don't change the name of the parent directory, in our example this is ruby, it can cause issues with the virtualenv unless you recreate it.
  • Similar to the warning above, don't change the name of the role in playbook.yml, it must match that of the parent directory.

Writing Our First Test

As with any test driven development we should start by writing some tests before we then make those test pass by writing our Ansible code. Molecule has already setup an example test file tests/test_default.py that we'll edit. First we'll test that Ruby's build dependencies are installed, in reality testing these is probably overkill but it'll serve as an example.

import testinfra.utils.ansible_runner

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    '.molecule/ansible_inventory').get_hosts('all')


def test_gcc_is_installed(host):
    pkg = host.package("gcc")

    assert pkg.is_installed

When running molecule test it wraps a bunch of molecule sub-commands that create a container, runs tests, and then destroys that container. We can speed up the feedback loop when developing by running the individual molecule sub-commands instead.

Create the Docker container like so:

(venv) $ molecule create
--> Creating instances...
--> Creating Ansible compatible image of amazonlinux:latest ...
Creating container ruby with base image amazonlinux:latest...
Container created.

We can then run our tests, we obviously expect them to fail at this point since we've not run any Ansible on the container.

(venv) $ molecule verify
--> Executing ansible-lint...
--> Executing flake8 on *.py files found in tests/...
--> Executing testinfra tests found in tests/...
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /home/rosstimson/Devops/ansible/ruby, inifile:
plugins: testinfra-1.6.3
collected 1 itemss

tests/test_default.py F

=================================== FAILURES ===================================
_____________________ test_gcc_is_installed[docker://ruby] _____________________

host = <testinfra.host.Host object at 0x7f3388947090>

    def test_gcc_is_installed(host):
	pkg = host.package("gcc")

>       assert pkg.is_installed
E       assert False
E        +  where False = <package gcc>.is_installed

tests/test_default.py:10: AssertionError
=============================== warnings summary ===============================
None
  Module already imported so can not be re-nwritten: testinfra

-- Docs: http://doc.pytest.org/en/latest/warnings.html
===================== 1 failed, 1 warnings in 0.52 seconds =====================
ERROR:

  RAN: /home/rosstimson/Devops/ansible/ruby/venv/bin/testinfra tests/test_default.py --connection=docker

  STDOUT:


  STDERR:

Write Ansible to Pass the Test

This test failure is exactly what we wanted since we've not written any Ansible to install the gcc package, we'll address that now. Create a new directory and file at tasks/main.yml.

---

- name: install ruby build dependencies
  yum:
    name: "{{ item }}"
  with_items:
    - gcc

I've pre-empted the fact we'll need a bunch of dependencies here by using a with_items loop. We can now do the Ansible run:

(venv) $ molecule converge
--> Starting Ansible Run...

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [ruby]

TASK [ruby : install ruby build dependencies] **********************************
changed: [ruby] => (item=[u'gcc'])

PLAY RECAP *********************************************************************
ruby                       : ok=2    changed=1    unreachable=0    failed=0

molecule converge  12.51s user 2.15s system 11% cpu 2:07.90 total

Verify our test is now passing:

(venv) $ molecule verify
--> Executing ansible-lint...
--> Executing flake8 on *.py files found in tests/...
--> Executing testinfra tests found in tests/...
============================= test session starts ==============================
platform linux2 -- Python 2.7.13, pytest-3.1.2, py-1.4.34, pluggy-0.4.0
rootdir: /home/rosstimson/Devops/ansible/ruby, inifile:
plugins: testinfra-1.6.3
collected 1 itemss

tests/test_default.py .

=============================== warnings summary ===============================
None
  Module already imported so can not be re-written: testinfra

-- Docs: http://doc.pytest.org/en/latest/warnings.html
===================== 1 passed, 1 warnings in 0.42 seconds =====================

Keeping Things DRY

We could write a test like this for each of our dependencies needed to build Ruby but that would be a lot of repetition. To DRY Don't Repeat Yourself. this up we can use Pytest Parametrizing. Note that in order to use this I've had to import pytest at the start of the file. Edit the tests/test_default.py again to include the rest of the necessary packages.

import testinfra.utils.ansible_runner
import pytest

testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    '.molecule/ansible_inventory').get_hosts('all')


@pytest.mark.parametrize("name", [
    ("automake"),
    ("bison"),
    ("gcc"),
    ("gdbm-devel"),
    ("libffi-devel"),
    ("libyaml-devel"),
    ("ncurses-devel"),
    ("openssl-devel"),
    ("readline-devel"),
    ("zlib-devel"),
])
def test_ruby_build_dependencies(host, name):
    pkg = host.package(name)
    assert pkg.is_installed

Go ahead and run molecule verify again to make sure the tests fail. We can then fix them with the following additions to the Ansible in tasks/main.yml.

---

- name: install ruby build dependencies
  yum:
    name: "{{ item }}"
  with_items:
    - automake
    - bison
    - gcc
    - gdbm-devel
    - libffi-devel
    - libyaml-devel
    - ncurses-devel
    - openssl-devel
    - readline-devel
    - zlib-devel

And again we can make sure our tests are now passing by converging and verifying again. This verify -> converge -> verify workflow is what I spoke about in the introduction.

Further Tests

Now that we've seen how you write basic tests and how the Molecule converge / verify workflow works we can flesh out our test suite by testing the Ruby executable which after all is the ultimate aim of this Ansible role. Modify your tests/test_default.py file to include the new tests.

import testinfra.utils.ansible_runner
import pytest
import re


testinfra_hosts = testinfra.utils.ansible_runner.AnsibleRunner(
    '.molecule/ansible_inventory').get_hosts('all')


@pytest.mark.parametrize("name", [
    ("automake"),
    ("bison"),
    ("gcc"),
    ("gdbm-devel"),
    ("libffi-devel"),
    ("libyaml-devel"),
    ("ncurses-devel"),
    ("openssl-devel"),
    ("readline-devel"),
    ("zlib-devel"),
])
def test_ruby_build_dependencies(host, name):
    pkg = host.package(name)
    assert pkg.is_installed


def test_ruby_executable_file(host):
    exe = host.file("/usr/local/bin/ruby")
    assert exe.user == "root"
    assert exe.group == "root"
    assert exe.mode == 0o755


def test_ruby_command(host):
    cmd = host.check_output("/usr/local/bin/ruby -v")
    assert re.match("^ruby 2.4.1*", cmd)

You can see we now test for the Ruby executable and make sure it has the expected owner and permissions. We also call the command and use a regular expression to make sure the stdout returns the version of Ruby I expect. Note the import re which allows us to use regex matching in the last test.

Build Ruby

Our tests now cover the Ruby install itself so we can now add to our Ansible code and make those tests pass. Edit the tasks/main.yml file to download the ruby source code and then build it from source, we can also abstract out some of the details into default variables to make the role more reusable.

---

- name: install ruby build dependencies
  package:
    name: "{{ item }}"
  with_items:
    - gcc
    - automake
    - zlib-devel
    - libyaml-devel
    - openssl-devel
    - gdbm-devel
    - readline-devel
    - ncurses-devel
    - libffi-devel

- name: download ruby source code tarball
  get_url:
    url: http://cache.ruby-lang.org/pub/ruby/{{ ruby_minor_version }}/ruby-{{ ruby_minor_version }}.{{ ruby_teenie_version }}.tar.gz
    dest: /tmp/ruby-{{ ruby_minor_version }}.{{ruby_teenie_version }}.tar.gz
    sha256sum: "{{ ruby_sha256sum }}"

- name: extract ruby source code tarball
  unarchive:
    src: /tmp/ruby-{{ ruby_minor_version }}.{{ ruby_teenie_version }}.tar.gz
    dest: /tmp
    copy: no

- name: compile and install ruby
  command: "{{ item }}"
  with_items:
    - ./configure --prefix={{ ruby_prefix }}
    - make
    - make install
  args:
    chdir: /tmp/ruby-{{ ruby_minor_version }}.{{ ruby_teenie_version }}
    creates: "{{ ruby_prefix }}/bin/ruby"

We will of course also need to supply these variable in a defaults file, do this by creating defaults/main.yml:

---

ruby_minor_version: 2.4
ruby_teenie_version: 1
ruby_sha256sum: a330e10d5cb5e53b3a0078326c5731888bb55e32c4abfeb27d9e7f8e5d000250
ruby_prefix: /usr/local

Testing Within Ansible Run Itself

As mentioned in the introduction, many people within the Ansible community believe it is uneccessary to write tests like we have been and that if needed you should just write your tests into the Ansible run itself. Although we've already used TestInfra to test our Ruby install let's look at how you might go about baking testing into the Ansible run. Back in the tasks/main.yml file add the following.

# Override the "changed" result, we are not changing anything here, just
# calling the ruby executable so we can check its version.
#
# Skip ansible-lint here as it will flag this as unecessary with the following:
# [ANSIBLE0012] Commands should not change things if nothing needs doing
- name: test ruby version
  command: "{{ ruby_prefix }}/bin/ruby -v"
  register: test_ruby_version
  changed_when: False
  tags:
    - skip_ansible_lint

- name: assert ruby version
  assert:
    that:
      - "'ruby {{ ruby_minor_version }}.{{ ruby_teenie_version }}' in test_ruby_version.stdout"

You can see this is very similar to what our TestInfra test does. It calls ruby -v at our installed location and makes sure the version we expect is shown in stdout. Note that we skip the ansible-lint by adding a tag, what we are doing here doesn't look quite right to ansible-lint since we are not making changes to the system, rather we are just calling a command to check its output. We also need to tell Ansible that even though this command will execute on all subsequent Ansible runs we are not actually making a change to the system. Without the changed_when override Ansible would report this as changed every time and the idempotence check would fail.

Refactor Tests for an Additional OS

Let's refactor our Ansible role so that it will work on both Amazon Linux and Ubuntu. It should actually work with many RHEL or Debian like operating systems once complete.

Refactoring like this is less daunting now that we have tests even when the Ansible code base starts getting larger and more complicated.

First off edit our molecule.yml to add in an extra container and give them sensible names to distinguish between the two.

docker:
  containers:
    - name: ruby-amazon
      image: amazonlinux
      image_version: latest
    - name: ruby-ubuntu
      image: ubuntu
      image_version: latest

The end result of our role is obviously to install Ruby so most of our tests will work no matter what the OS. However, Debian based systems have different package names for the build dependencies which we need to accomodate for. First let's modify our tests to check for specific packages depending on the OS. In order to separate out our OS specific tests, create a new directory and file at tests/amazon/test_amazon.py with the following.

import pytest

testinfra_hosts = ["ruby-amazon"]


@pytest.mark.parametrize("name", [
    "automake",
    "bison",
    "gcc",
    "gdbm-devel",
    "libffi-devel",
    "libyaml-devel",
    "ncurses-devel",
    "openssl-devel",
    "readline-devel",
    "zlib-devel",
])
def test_amazon_ruby_build_dependencies(host, name):
    pkg = host.package(name)
    assert pkg.is_installed

We need to create a similar file to test for the different package names on Ubuntu at /tests/ubuntu/test_ubuntu.py:

import pytest

testinfra_hosts = ["ruby-ubuntu"]


@pytest.mark.parametrize("name", [
    "bison",
    "build-essential",
    "libffi-dev",
    "libgdbm-dev",
    "libncurses5-dev",
    "libreadline-dev",
    "libssl-dev",
    "libyaml-dev",
    "zlib1g-dev",
])
def test_ubuntu_ruby_build_dependencies(host, name):
    pkg = host.package(name)
    assert pkg.is_installed

Remove the similar section from tests/test_default.py as these are the tests that will run on both containers / operating systems; also remove the import pytest line as it is not used now and it will be picked up by the linter.

For simpler tests where you don't want to divide things out into different files you could use something like this:

def test_command(host):
    os = host.system_info.distribution

    if os in ("ubuntu", "debian"):
        assert host.exists("apt")
    elif os in ("amzn", "centos", "fedora"):
        assert host.exists("yum")

Update Ansible for Debian Based Systems

Now that our tests now work with both Amazon Linux (RHEL-like) and Ubuntu (Debian-like) we need to modify our Ansible slightly to cater for the differing package names. We can achieve this by using Ansible's when conditional, in tasks/main.yml replace our current ruby dependencies task with two.

# RHEL-like operating systems such as Amazon Linux.
- name: install ruby build dependencies
  package:
    name: "{{ item }}"
  with_items:
    - automake
    - bison
    - gcc
    - gdbm-devel
    - libffi-devel
    - libyaml-devel
    - ncurses-devel
    - openssl-devel
    - readline-devel
    - zlib-devel
  when: ansible_os_family == "RedHat"

# Debian-like operating systems such as Ubuntu.
- name: install ruby build dependencies
  package:
    name: "{{ item }}"
  with_items:
    - bison
    - build-essential
    - libffi-dev
    - libgdbm-dev
    - libncurses5-dev
    - libreadline-dev
    - libssl-dev
    - libyaml-dev
    - zlib1g-dev
  when: ansible_os_family == "Debian"

We should be able to do a molecule test again and all tests should pass but this time we spin up two different OS containers simultaneously. Make sure to destroy any current running containers with molecule destroy if you've not done so since modifying molecule.yml.

Abstract OS Packages into Variables

I know that a lot of people would cleanup these two tasks by setting the list of packages in a variable instead. Just to illustrate how writing tests gives us the confidence to refactor as much as we like let's try doing just that by creating a vars/main.yml with the following content.

---

redhat_pkg:
  - automake
  - bison
  - gcc
  - gdbm-devel
  - libffi-devel
  - libyaml-devel
  - ncurses-devel
  - openssl-devel
  - readline-devel
  - zlib-devel

debian_pkg:
  - bison
  - build-essential
  - libffi-dev
  - libgdbm-dev
  - libncurses5-dev
  - libreadline-dev
  - libssl-dev
  - libyaml-dev
  - zlib1g-dev

Now amend our tasks in tasks/main.yml to use these variables.

# RHEL-like operating systems such as Amazon Linux.
- name: install ruby build dependencies
  package:
    name: "{{ item }}"
  with_items: "{{ redhat_pkg }}"
  when: ansible_os_family == "RedHat"

# Debian-like operating systems such as Ubuntu.
- name: install ruby build dependencies
  package:
    name: "{{ item }}"
  with_items: "{{ debian_pkg }}"
  when: ansible_os_family == "Debian"

Confirm all is well by running the test suite again.

Conclusion and Next Post

I hope this post has got you interested in testing your Ansible code more and has shown how easy it is to get started. Although this does not seem widely done in the Ansible community personally I really like this way of working and testing can only improve the Ansible roles I write.

In my next post I hope to write about putting our Ruby role into some sort of CI build tool so that our tests run automatically when pushed to a Git repository. I then want to explore how larger wrapper type roles that pull in other roles from various sources such as Ansible Galaxy or Github can be used and put into a CI/CD pipeline.

Footnotes:

1

You can actually use Test Kitchen to test other configuration management tools such as Ansible, Salt, Puppet etc.

2

Molecule can use Serverspec for testing but it would require the installation of Ruby. Ansible itself is written in Python so let's stick with Python tools.

3

I had issues with Python 3 when I tried, YMMV.

4

A collaborator can use the requirements file with: $ pip install -r requirements.txt.

5

Don't Repeat Yourself.

6

It should actually work with many RHEL or Debian like operating systems once complete.

Contact Info