TripleO(-Ansible) Bootcamp

Kevin Carter -

In this post, I intend to bring new folks into TripleO-Ansible by explaining how TripleO is evolving. We're building a modular set of Ansible roles, playbooks, plugins, and scripts, which are reforming the foundation of enterprise-ready OpenStack environments. While that's awesome in its own right, these tools are also built to provide easy to consume utilities which can make day to day operations and system administration easier than ever.

While I've covered role development in detail before, this post is a recap from the Red Hat TripleO-Bootcamp session I presented in Brno. The content within this post was developed to show how straightforward it is to create, test, and document a role. Once the role is ready, this post goes into consuming the newly created role and extending playbooks. Notice, with all that fancy alliteration, I never said anything about TripleO. While TripleO-Ansible is part of the OpenStack TripleO project, the roles, plugins, and playbooks we're developing are useful on their own and are great for general system administration and operations. While the primary focus is deploying OpenStack clouds, it is not the only purpose. This means our tooling has the added benefit of being a modular Ansible collection with the goal of increasing the stability of any application deployment, not just Enterprise-ready OpenStack.

This post aims to be tactical and starts from a lab environment that is running RHEL8 or CentOS8. The development environment doesn't need much in the way or RAM or storage; a VM (KVM, VirtualBox, LXC/D) with 2GiB of RAM, 8 GiB of storage, and 1 vCPU would be more than enough to perform all of the tasks. At the end of the post, you should be able to build roles and integrate them into playbooks with absolute ease. You will also understand the testing process and how things are auto-documented.

Developing a new role

Developing a new role is simple in TripleO-Ansible. The TripleO-Ansible software developer kit, within the code repository, provides for everything needed to generate documentation, tests, and the role wire-frame. TripleO-Ansible uses a playbook to create new roles, add the new role to the ZuulCI job configuration, and document the new role.

Running the role generation playbook

To run the role generation playbook, run ten simple commands.

  • Install package dependencies.
sudo yum install gcc python*-virtualenv
  • Ensure the python executable is available (optional)
if [[ ! -f /usr/bin/python ]]; then
  sudo ln -s /usr/bin/python3 /usr/bin/python
fi
  • Create a new python virtual environment for the molecule tests.
python3 -m virtualenv ~/test-python
  • With a clean development environment, clone the TripleO-Ansible repository.
git clone https://opendev.org/openstack/tripleo-ansible ~/tripleo-ansible
  • Change directory to the newly cloned repository.
cd ~/tripleo-ansible
  • Install all of the python requirements into the virtual environment.
~/test-python/bin/pip install ansible
  • Create a new branch.
git checkout -b tripleo-hello-world
  • Activate the venv.
source ~/test-python/bin/activate
  • Generate a new role.
~/test-python/bin/ansible-playbook -i localhost, role-addition.yml -e role_name=tripleo-hello-world
  • Deactivate the venv
deactivate

This playbook will generate the new role skeleton, create the documentation page for the new role, and add the new role to the ZuulCI configuration. The developer can then change directory to the newly generated role, ~/tripleo-ansible/tripleo_ansible/roles/tripleo-hello-world, and begin work.

Testing roles within TripleO-Ansible

Running a simple test within TripleO-Ansible requires nothing more than a vacant OS running RHEL-8, CentOS-8, or Fedora-28+. Simple test instances as provided by libvirt, vagrant, virtualbox, etc. are more than enough to run tests with a clean system.

The TripleO-Ansible project is gated with ZuulCI, and all code review is done within Gerrit. OpenStack Infrastructure manages the code review portal and testing infrastructure.

Running local, like ZuulCI

Testing TripleO-Ansible roles can be performed locally in the same way as ZuulCI. This functionality gives developers a simple way to develop new role functionality on local test instances, which allows developers the ability to create content with confidence that it will pass the gate.

Breaking down role tests

Every role within TripleO-Ansible is tested independently from every other role. While the code repository is a monolith, the testing framework is anything but. Tests are broken down into two categories, molecule, and scenario.

Molecule tests are triggered anytime a change is made to a given role, and every role has its own specific set of molecule tests.

Integration tests run a full deployment, ensuring we're maintaining functionality and system integration within the deployment framework.

Whenever a review is submitted to the TripleO-Ansible project, ZuulCI matches the change to a limited set of jobs. The job matching functionality ensures we're testing what is necessary to provide developer feedback and give reviews high confidence the being reviewed is sound. It is expected that a single review submitted to a single role will trigger at least one molecule job and one integration job.

If a review spans more than one role, multiple jobs may be triggered.

Molecule testing overview

Molecule tests are built to exercise every code path within a role. Each role may have one or more molecule scenarios that run various code path's within a role. At a minimum, each role will have the default scenario. This scenario runs through the default role code path.

Integration testing overview

The TripleO-CI maintains integration tests. TripleO-Ansible uses the integration jobs to ensure no review breaks expected functionality. These jobs will run an end to end deployment, using either multi-node or standalone test environments.

Running local tests

Running local tests is simple and can be done with as little as two commands.

  • Change directory to the newly cloned repository.
cd ~/tripleo-ansible
  • Run the local test script with the name of a given role being tested.
./scripts/run-local-test tripleo-hello-world

The script will run all molecule tests with a given role, in the same way ZuulCI would. This interface gives developers a simple means to test roles and role changes easily.

Running manual tests

Tests can be run manually using molecule from within the role directory. This functionality is considered "advanced," as it requires the role developer to know more about the test interactions and the molecule CLI utility. While "advanced," it is beneficial for debugging specific parts of a test run and can produce fascinating results.

Running molecule tests requires a little more setup when running from a clean workstation, however it can be done in as little as twelve steps.

  • With a clean development environment, clone the TripleO-Ansible repository.
This command is not required if there's already a checkout on the local
workstation.
git clone https://opendev.org/openstack/tripleo-ansible ~/tripleo-ansible
  • Install package dependencies.
yum install gcc python*-virtualenv
  • Create a new python virtual environment for the molecule tests.
python3 -m virtualenv ~/test-python
  • Install all of the python requirements into the virtual environment.
~/test-python/bin/pip install -r ~/tripleo-ansible/requirements.txt -r ~/tripleo-ansible/molecule-requirements.txt
  • Change your user to yourself

You can also log out and log back in.

sudo su - $USER

This is only required because our user will need access to the Docker process.

  • Activate the test-python virtual environment.
source ~/test-python/bin/activate
  • Install any system packages using the bindep installation script.
~/tripleo-ansible/scripts/bindep-install
  • Source the test ansible environment variables.
source ~/tripleo-ansible/ansible-test-env.rc
  • Change directory to the role within the newly cloned repository.
cd ~/tripleo-ansible/tripleo_ansible/roles/tripleo-hello-world
  • Run the molecule test command.
molecule test --all
  • Get coffee

The molecule test command executes all tests within a given role, giving direct feedback about the role to the developer. While the process is longer than running the local test script, this method gives developers access to the test functions of a role, which can be very useful when debugging hard issues, or developing new tests.

  • Deactivate the venv
deactivate
Testing review

When testing with molecule, the test should aim to be quick and straightforward. An example molecule test should finish in as little as 1 minute. Depending on what is being tested, a molecule run could take up to 45 minutes; this is the ZuulCI upper limit. In general, if the test scenario takes longer than 45 minutes, the tests should be broken down into smaller molecule scenarios. If smaller scenarios are not possible, tests should be performed using the integration test suit, which has a time limit of up to 3 hours.

Checking for lint

Lint checking in TripleO-Ansible is all run through the OpenStack standard tox. To run lint checks on the code base simply invoke tox with the linters environment option. While tox can be installed in any number of ways, it is recommended to create a new virtual environment to ensure you have the latest tox installed prior to running the tests.

The following six steps will do everything needed to lint check the code base.

  • With a clean development environment, clone the TripleO-Ansible repository.

This command is not required if there's already a checkout on the local
workstation.

git clone https://opendev.org/openstack/tripleo-ansible ~/tripleo-ansible
  • Create a new virtual environment.
python3 -m virtualenv ~/tox
  • Install tox.
~/tox/bin/pip install tox
  • Activate the virtual environment.
source ~/tox/bin/activate
  • Change directory to the newly cloned repository.
cd ~/tripleo-ansible
  • Run the linters test.
tox -e linters

The linters check will validate the code-bases passes our coding standards. This includes yaml-lint, ansible-lint, flake8, bashate, and a check for any broken symlinks. These tests take a couple minutes but will ensure that there are not known syntax issues being introduced into the code-base.

Documentation

In the case of new role development, the documentation source code is contained under, doc/source/roles/. All role documentation files are prefixed with role-. As mentioned, the auto-documentation plugin will also parse modules. The source code for the module documentation is contained under doc/source/modules/, and uses a prefix of the plugin type, (action-, callback-, modules-, etc.).

Creating documentation

Creating documentation is simple and requires only a simple shim to get
started.

  • Create the documentation reStructuredText file.

This command is not required if the file already exists.

cat > doc/source/roles/role-tripleo-hello-world.rst <<EOF
==========================
Role - tripleo-hello-world
==========================

.. ansibleautoplugin::
   :role: tripleo_ansible/roles/tripleo-hello-world
EOF
  • With the file in place, edit the file to add additional information to the role documentation as needed.

As you may notice, there is an autodoc plugin being used to create the role documentation. While extremely powerful, this plugin is not the only source of documentation developers should rely on. If additional information is required or useful to the end-user, additional documentation can be added to any role in reStructuredText format.

Generating the documentation

When submitting changes to TripleO-Ansible, ZuulCI will generate documentation anytime files within the docs/ path are changed. Documentation is generated using the OpenStack standard tox through sphinx. To ease in the creation of documentation the TripleO-Ansible project employees a sphinx plugin which will parse Ansible roles and modules, and generate documentation from it's derived information.

  • Run the docs generation.
tox -e docs
  • Change directory to the built html documentation.
cd ~/tripleo-ansible/doc/build/html
  • Get your server IP address
ip -o r g 1 | awk '{print $7}'
  • Start a simple html server so we can browse the documentation.

You may need to add a firewall rule sudo iptables -I INPUT 1 -p tcp --dport 12345 -j ACCEPT.

python -m http.server 12345 --bind $(ip -o r g 1 | awk '{print $7}')

Open a browser window and navigate to your servers IP address on port 12345.

Stop the simple http server with CTRL-C.

Deactivate the venv

deactivate

Create a new playbook for our role

The roles we create can be used as-is. All of the roles we're building can be used without the TripleO deployment framework. This means we can use our  roles in playbooks to operate systems, deploy services, or otherwise run tasks; effectively enabling operators, developers, and deployers to make use of ansible, directly.

Building a playbook is simple.

  • Create a playbook file referencing the role.
cat > ~/tripleo-ansible/tripleo_ansible/playbooks/tripleo-hello-world.yaml <<EOF
---
- hosts: localhost
  connection: local
  roles:
    - role: tripleo-hello-world
EOF
  • Activate the test-python virtual environment.
source ~/test-python/bin/activate
  • Source the test ansible environment variables.
source ~/tripleo-ansible/ansible-test-env.rc
  • Run our new playbook
ansible-playbook -i 'localhost,' ~/tripleo-ansible/tripleo_ansible/playbooks/tripleo-hello-world.yaml

Given we can see our role works, lets make our playbook do something interesting. We'll set the timezone of our host to UTC. To do this we'll run the tripleo-timezone role in our new playbook.

  • Update our new playbook.
---
- hosts: localhost
  connection: local
  roles:
    - role: tripleo-hello-world
    - role: tripleo-timezone
      tripleo_timezone: UTC
  • Re-run our new playbook
ansible-playbook -i 'localhost,' ~/tripleo-ansible/tripleo_ansible/playbooks/tripleo-hello-world.yaml

Once again, lets update our playbook to make it do more with our existing
roles. Lets setup the SSH config, enable a message of the day and create
a banner.

  • Update our new playbook.
---
- hosts: localhost
  connection: local
  roles:
    - role: tripleo-hello-world
    - role: tripleo-timezone
      tripleo_timezone: UTC
    - role: tripleo-sshd
      tripleo_sshd_motd_enabled: true
      tripleo_sshd_banner_enabled: true
  • Once again re-run our new playbook.
ansible-playbook -i 'localhost,' ~/tripleo-ansible/tripleo_ansible/playbooks/tripleo-hello-world.yaml

That's all folks!

I hope this bootcamp recap has been helpful. If you have any questions please reach out to the group in the #tripleo channel on freenode. There's an incredible, and inviting community there building the future of Enterprise ready OpenStack clouds and we would love you to join us as we evolve into the next phase.