/ nspawn

OpenStack-Ansible machinectl image management

nspawn | systemd | containers | images | machinectl | openstack-ansible | openstack | osa

With recent changes in OpenStack-Ansible we're now able to import and export container images for using the built-in systemd tooling via machinectl. This gives us the ability to rapidly provision or replace containers as needed from a pre-seeded source. In this post, I'll cover how an operator might export a container and then import a container from both a local cache as well as a remote one.

The Setup

Assume you have an OpenStack-Ansible deployment running the latest release of Pike and have set the container storage options to use the new machinectl backend. The available storage options are covered within the role documentation as found here. With the setting lxc_container_backing_store set to machinectl all of the containers will be stored and managed by systemd. The machinectl toolchain provides a lot of capabilities however in this post we're only going to cover export and import of a container image.

Imagine the environment

Within a container host we can login and list all of the available container images.

# machinectl list-images
NAME                                            TYPE      RO  USAGE  CREATED                     MODIFIED
compute1_aodh_container-f08d2204                subvolume no  1.1G   Thu 2017-10-26 17:03:13 CDT n/a     
compute1_ceilometer_central_container-56318e0e  subvolume no  1.0G   Thu 2017-10-26 17:03:19 CDT n/a     
compute1_cinder_api_container-86552654          subvolume no  1.2G   Thu 2017-10-26 17:39:03 CDT n/a     
compute1_cinder_scheduler_container-2c0c6061    subvolume no  1.2G   Thu 2017-10-26 17:03:08 CDT n/a     
compute1_galera_container-53877d98              subvolume no  1.1G   Thu 2017-10-26 17:03:18 CDT n/a     
compute1_glance_container-78b73e1a              subvolume no  2.1G   Thu 2017-10-26 17:39:00 CDT n/a     
compute1_gnocchi_container-9a4b182b             subvolume no  432.0M Thu 2017-10-26 17:39:04 CDT n/a     
compute1_heat_apis_container-c973ef5a           subvolume no  1.1G   Thu 2017-10-26 17:03:18 CDT n/a     
compute1_heat_engine_container-ae51062c         subvolume no  1.1G   Thu 2017-10-26 17:03:12 CDT n/a     
compute1_horizon_container-4148c753             subvolume no  1.3G   Thu 2017-10-26 17:39:05 CDT n/a     
compute1_keystone_container-7a0a3834            subvolume no  1.3G   Thu 2017-10-26 17:39:09 CDT n/a     
compute1_memcached_container-782a6588           subvolume no  495.7M Thu 2017-10-26 17:03:11 CDT n/a     
compute1_neutron_agents_container-de8a4d37      subvolume no  1.1G   Thu 2017-10-26 17:03:13 CDT n/a     
compute1_neutron_server_container-219f00f7      subvolume no  1.1G   Thu 2017-10-26 17:03:07 CDT n/a     
compute1_nova_api_metadata_container-9a8fe9ae   subvolume no  1.4G   Thu 2017-10-26 17:03:17 CDT n/a     
compute1_nova_api_os_compute_container-2a4faa2c subvolume no  1.4G   Thu 2017-10-26 17:39:03 CDT n/a     
compute1_nova_api_placement_container-42904e4c  subvolume no  1.4G   Thu 2017-10-26 17:39:05 CDT n/a     
compute1_nova_conductor_container-5109b386      subvolume no  1.4G   Thu 2017-10-26 17:03:07 CDT n/a     
compute1_nova_console_container-cf223830        subvolume no  1.4G   Thu 2017-10-26 17:39:00 CDT n/a     
compute1_nova_scheduler_container-832bf438      subvolume no  1.4G   Thu 2017-10-26 17:03:08 CDT n/a     
compute1_rabbit_mq_container-652f0bda           subvolume no  842.2M Thu 2017-10-26 17:39:08 CDT n/a     
compute1_repo_container-754d214c                subvolume no  1.3G   Thu 2017-10-26 17:03:12 CDT n/a     
compute1_swift_proxy_container-fb47a052         subvolume no  1.0G   Thu 2017-10-26 17:03:07 CDT n/a     
ubuntu-xenial-amd64                             subvolume no  369.6M Mon 2017-10-16 11:17:19 CDT n/a

These container images all show the name and size of the image as well the type. The names of these container images all correspond to our running LXC containers as seen with a simple list.

# lxc-ls -f 
NAME                                            STATE   AUTOSTART GROUPS            IPV4
compute1_aodh_container-f08d2204                RUNNING 1         onboot, openstack 10.0.3.118, 172.16.26.75
compute1_ceilometer_central_container-56318e0e  RUNNING 1         onboot, openstack 10.0.3.134, 172.16.26.190
compute1_cinder_api_container-86552654          RUNNING 1         onboot, openstack 10.0.3.34, 172.16.26.86
compute1_cinder_scheduler_container-2c0c6061    RUNNING 1         onboot, openstack 10.0.3.66, 172.16.26.153
compute1_galera_container-53877d98              RUNNING 1         onboot, openstack 10.0.3.137, 172.16.26.208
compute1_glance_container-78b73e1a              RUNNING 1         onboot, openstack 10.0.3.199, 172.16.26.241
compute1_gnocchi_container-9a4b182b             RUNNING 1         onboot, openstack 10.0.3.225, 172.16.26.219
compute1_heat_apis_container-c973ef5a           RUNNING 1         onboot, openstack 10.0.3.253, 172.16.26.236
compute1_heat_engine_container-ae51062c         RUNNING 1         onboot, openstack 10.0.3.8, 172.16.26.132
compute1_horizon_container-4148c753             RUNNING 1         onboot, openstack 10.0.3.194, 172.16.26.49
compute1_keystone_container-7a0a3834            RUNNING 1         onboot, openstack 10.0.3.133, 172.16.26.146
compute1_memcached_container-782a6588           RUNNING 1         onboot, openstack 10.0.3.18, 172.16.26.126
compute1_neutron_agents_container-de8a4d37      RUNNING 1         onboot, openstack 10.0.3.150, 172.16.26.220
compute1_neutron_server_container-219f00f7      RUNNING 1         onboot, openstack 10.0.3.87, 172.16.26.57
compute1_nova_api_metadata_container-9a8fe9ae   RUNNING 1         onboot, openstack 10.0.3.101, 172.16.26.170
compute1_nova_api_os_compute_container-2a4faa2c RUNNING 1         onboot, openstack 10.0.3.49, 172.16.26.116
compute1_nova_api_placement_container-42904e4c  RUNNING 1         onboot, openstack 10.0.3.158, 172.16.26.115
compute1_nova_conductor_container-5109b386      RUNNING 1         onboot, openstack 10.0.3.61, 172.16.26.101
compute1_nova_console_container-cf223830        RUNNING 1         onboot, openstack 10.0.3.139, 172.16.26.123
compute1_nova_scheduler_container-832bf438      RUNNING 1         onboot, openstack 10.0.3.92, 172.16.26.199
compute1_rabbit_mq_container-652f0bda           RUNNING 1         onboot, openstack 10.0.3.173, 172.16.26.54
compute1_repo_container-754d214c                RUNNING 1         onboot, openstack 10.0.3.185, 172.16.26.80
compute1_swift_proxy_container-fb47a052         RUNNING 1         onboot, openstack 10.0.3.41, 172.16.26.247
compute1_utility_container-cbd7b73e             RUNNING 1         onboot, openstack 10.0.3.97, 172.16.26.94

To exercise an export and import operation I'm using the compute1_utility_container-cbd7b73e container however the steps covered within this post can apply to ANY container on the system.

The covered capabilities within this post do very little for the upgrade story at this point.

Container Export

Exporting a containers image is simple. There's nothing you need to do on the host to initiate the process and there's no setup within the container to prepare for export. To perform a basic export simply run the following command:

# machinectl export-tar compute1_utility_container-cbd7b73e utility_container.tar.gz`

The command will create a compressed archive of the running container. This task will take a minute or two resulting in a local tarball which can be used later.

Notice About Container Exports

Container exports WILL NOT create an archive of anything bind mounted within the container runtime. This means externally mounted bits like databases and repository caches will not be part of the archive. The point of a container export is to provide the building blocks of a service and is not a replacement for backups.

Container Import

In this example, I'm going to replace a running utility container with a different storage backend. Before importing a container image into the environment we need to make sure the container is not running. Once we've ensured the container is STOPPED we'll clean up the old image and replace it using one of two methods; import-tar or pull-tar. This operation will keep the existing LXC container definition making it possible to run these commands within a live environment without having to rerun any of the OSA playbooks.

Stopping the container

To stop the container we'll use the lxc-stop command.

# lxc-stop -n compute1_utility_container-cbd7b73e

Now verify the container is STOPPED.

# lxc-info -n compute1_utility_container-cbd7b73e
Name:           compute1_utility_container-cbd7b73e
State:          STOPPED

Remove the old container storage

Validate the old image exists.

# machinectl show-image compute1_utility_container-cbd7b73e

Remove the image

# machinectl remove compute1_utility_container-cbd7b73e

After the image has been removed you can validate it's really gone with the show-image function again or you can move on to importing the image.

Import from a local file

Assuming the new image is on the local machine, run the following command to import the container image.

# machinectl import-tar utility_container.tar.gz compute1_utility_container-cbd7b73e

After a few seconds, the container image will be ready for use.

Import from a remote file

Assuming the new image stored on a remote server which is providing access to the images via HTTP(S), run the following command to import the container image.

# machinectl --verify=no pull-tar http://lab-lb01:8181/images/ubuntu/pike/amd64/utility_container.tar.gz compute1_utility_container-cbd7b73e 

After a minute or two, the container image will be ready for use.

Notice About Remote Import

When downloading an image from a remote server the ability to verify a container image is available as ether a SHA256 Sum and/or a GPG signature. In my example, I've disabled verification, however, this is not ideal especially in production. You can read more about the validation options here.

Restarting the container

Now that the container storage is replaced all that's left to do is restart the container. To do this we'll simply use the lxc-start command on the same container we stopped earlier.

# lxc-start -dn compute1_utility_container-cbd7b73e

Once started, verify the container is active and online using the lxc-info command.

# lxc-info -n compute1_utility_container-cbd7b73e
Name:           compute1_utility_container-cbd7b73e
State:          RUNNING
PID:            27243
IP:             10.0.3.97
IP:             172.16.26.94
CPU use:        0.74 seconds
BlkIO use:      68.00 KiB
Memory use:     22.69 MiB
KMem use:       8.99 MiB
Link:           cbd7b73e_eth0
 TX bytes:      520 bytes
 RX bytes:      637 bytes
 Total bytes:   1.13 KiB
Link:           cbd7b73e_eth1
 TX bytes:      488 bytes
 RX bytes:      446 bytes
 Total bytes:   934 bytes

If everything started and accessible over the network, your done.

Recap & Looking forward

Now, this simple example is great when running a simple update or replace however this same basic concept could also be applied to a greenfield deployment making the initial setup and configuration of containerized systems faster.

Imagine populating the OSA inventory and then dropping a complete set of container images from a remote source before running openstack-ansible setup-everything.yml.

Here's a pseudo playbook to gather facts on our container hosts and then pre-seed all of the container images; debug is being used to illustrate the command without actually running anything. With something as simple as this playbook we could ingest all of the required container images. Once this playbook was complete, we'd simply run the rest of the deployment like normal.

- name: Gather lxc container host facts
  hosts: "lxc_hosts"
  gather_facts: true

- name: Pull container images into the deployment
  hosts: "all_containers"
  gather_facts: false
  tasks:
    # Debug used to illustrate the command we'd run.
    - name: Pre-seed container images
      debug: 
        msg: >-
          machinectl --verify=no pull-tar http://lab-lb01:8181/images/{{ physical_distro }}/pike/{{ physical_arch }}/{{ inventory_hostname.split('_', 1)[-1].split('-')[0] }}.tar.gz {{ inventory_hostname }}
      delegate_to: "{{ physical_host }}"
  vars:
    physical_distro: "{{ hostvars[physical_host]['ansible_distribution'] }}"
    physical_arch: "{{ hostvars[physical_host]['ansible_architecture'] }}"

Having the ability to pre-seed container images should help ensure a consistent initial deployment and massively improve the speed of greenfield deployments. Best of all, we wouldn't have to sacrifice any capabilities. Almost everything OpenStack-Ansible deploys is within a container and with pre-seeded images, those bits would already be on disk simply making the playbooks a means to reconfigure an environment.

Just as we can import a series of containers we can also archive all of the containers from a running deployment. Being that all we need is the base container images, we could use something as simple as an All-In-One and archive the running containers. The resulting archive would then serve as the basis for all version controlled deployments (AIO or Multi-Node). Here's a pseudo playbook to archive all of the running containers; debug is being used to illustrate the command without actually running anything.

- name: Archive all containers from a deployment
  hosts: "all_containers"
  gather_facts: false
  tasks:
    - name: Archive container images
      debug: 
        msg: >-
          machinectl export-tar {{ inventory_hostname }} {{ inventory_hostname.split('_', 1)[-1].split('-')[0] }}.tar.gz
      delegate_to: "{{ physical_host }}"

The covered capabilities within this post do very little for the upgrade story at this point. That said, our more advanced storage options will give us new capabilities to explore and should help shape the conversation within the community so we can make informed decisions on how we make containerization and images work for us.

That's all folks

It's my hope that this simple post shines a light on some of the core technologies OpenStack-Ansible deployers already have access to and how our project is evolving as we move forward as a community.