/ nspawn

Networkd for nspawn with OpenStack-Ansible

Within this post, I will cover the systemd-networkd aspect of the PR to create a systemd-nspawn driver within OSA. This post will be working with systemd-networkd and systemd-resolved to integrate nspawn containers within the existing topology of a typical OpenStack-Ansible deployment.

Getting systemd-networkd working for you

macvlan-osa-prod-simple

One of the container interfaces, overview covered here,
has a single MACVLAN interface which binds to all containers. This single interface (Container Bridge) allows for external communication from the containers through the gateway using IP forwarding and masquerading. This special network is bound only to this one host, setups up a locally scoped DHCP network and pushes NTP and DNS to the containers from the host. To make this possible we're going to use two additional systemd built-ins called systemd-networkd and systemd-resolved.

Setting up the resolved

The systemd-resolved setup is nothing special and, in the test case, I used nothing more than the package installed defaults. See more about the options of systemd-resolved here. To setup systemd-resolved we simply link the appropriate file to our resolve.conf, enable the service and start it.

ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf  # Create the link
systemctl enable systemd-resolved.service  # Enable the service
systemctl start systemd-resolved.service  # Start the service

Building the Container Bridge

For the DHCP configuration we setup a simple dummy interface and a standalone bridge called br-container which will all be done using systemd-networkd.

Link configuration

The first thing we need is a link file, /etc/systemd/network/99-default.link, which will ensure all interfaces managed by systemd-networkd use a persistent mac address:

[Link]
MACAddressPolicy=persistent
Dummy Interface

Once the link file is in place, create a network device file for our dummy interface, /etc/systemd/network/00-dummy0.netdev, with the following contents:

[NetDev]
Name=dummy0
Kind=dummy

With the network device in place, we'll create a network configuration file, /etc/systemd/network/00-dummy0.network, so that the dummy0 interface is enrolled within the br-container device:

[Match]
Name=dummy0

[Network]
Bridge=br-container
Bridge Interface

Now that the dummy network device is ready, we're going create a network device file for our bridge, /etc/systemd/network/br-container.netdev, with the following contents:

[NetDev]
Name=br-container
Kind=bridge

Finally, we'll create our bridge network file, /etc/systemd/network/br-container.network, with all of our options to enable all of our needed options:

[Match]
Name=br-container

[Network]
Address=10.0.3.2/24
Gateway=10.0.3.1
IPForward=yes
IPMasquerade=yes
DHCPServer=yes

[DHCPServer]
PoolOffset=25
PoolSize=100
DefaultLeaseTimeSec=3h
EmitTimezone=yes
EmitNTP=yes
EmitDNS=yes
Load the new configs

Now that we have all of the needed files in place we can enable and start the systemd-networkd service:

systemctl enable systemd-networkd.service  # Enable the service
systemctl start systemd-networkd.service  # Start the service

Check the status of the systemd-networkd to make sure it's running:

systemctl status systemd-networkd.service  # Check service status

Finally, list out all of the networks managed:

networkctl list
IDX LINK             TYPE               OPERATIONAL SETUP     
  1 lo               loopback           carrier     unmanaged 
  2 ens3             ether              routable    unmanaged 
  3 br-container     ether              routable    configured
  4 dummy0           ether              degraded    configured

Note: the OPERATIONAL state of these interfaces will always be "degraded". This state is simply because the underlying interface at the lowest level is a dummy device. If you wanted br-container to be routable you could use an actual interface, in my case ens3, instead of the dummy device.

This simple bridge interface managed by systemd-resolved replaces a several iptables rules, a dnsmasq process with a specialized configuration limiting its scope to only the one host, and an ad-hoc interface running pre-up scripts gluing everything needed together. As you can surmise, this process can be fragile and should it get out of sync, it impacts every running container on a given host. With the new systemd-networkd capabilities we only have one process to manage in a single unified way across all supported operating systems. While this change may confuse operators initially due to the generally smaller footprint, folks will come to love this change as there's no longer any magic required to understand how this all works and recovering from failure is as simple as a service restart.

Connecting bridges to MacVlan interfaces

Now that we've setup our basic container bridge, which is capable of issuing addresses on our internal only network and routing that traffic out of our host, we're going to create a macvlan interface and wire it back into our the br-container device. To do this we're going to create a few more files.

MacVlan Interface

First create a network device file, /etc/systemd/network/mv-container.netdev for our mv-container interface:

[NetDev]
Name=mv-container
Kind=macvlan

[MACVLAN]
Mode=bridge

Then create a network file, /etc/systemd/network/mv-container.network, to ensure our mv-container is a link only network:

[Match]
Name=mv-container

[Network]
DHCP=no
MacVlan to Bridge Network Integration

Now create a veth pair file, /etc/systemd/network/mv-container-vs.netdev, which will be used to link the mv-container interface to the br-container interface:

[NetDev]
Name=mv-container-v1
Kind=veth

[Peer]
Name=mv-container-v2

Plugin the one side of the veth pair into the br-container interface by creating a network file, /etc/systemd/network/mv-container-v1.network, for this side of the veth:

[Match]
Name=mv-container-v1

[Network]
Bridge=br-container

And then Plugin the other one side of the veth pair into the mv-container interface by creating a network file, /etc/systemd/network/mv-container-v2.network, for the remaining side of the veth:

[Match]
Name=mv-container-v2

[Network]
MACVLAN=mv-container
Load the config and connect the networks

Finally, restart systemd-networkd and then list out all managed systemd-networkd interfaces again and see that the new mv-container interface is up configured:

systemctl restart systemd-networkd.service  # Restart the service
networkctl list

IDX LINK             TYPE               OPERATIONAL SETUP     
  1 lo               loopback           carrier     unmanaged 
  2 ens3             ether              routable    unmanaged 
  3 br-container     ether              routable    configured
  4 dummy0           ether              degraded    configured
  5 mv-container-v2  ether              degraded    configured
  6 mv-container-v1  ether              degraded    configured
  7 mv-container     ether              degraded    configured

Integrating the rest of the network interfaces

In a typical OpenStack-Ansible deployment we will have multiple purpose built bridges which will all need macvlan integration. The process to get these bridges and macvlan devices squared away can be taken care of by running an easy script:

#!/usr/bin/env bash
# Create the macvlan network device
cat >/etc/systemd/network/mv-${1}.netdev <<EOF
[NetDev]
Name=mv-${1}
Kind=macvlan

[MACVLAN]
Mode=bridge
EOF

# Create the macvlan network
cat >/etc/systemd/network/mv-${1}.network <<EOF
[Match]
Name=mv-${1}

[Network]
DHCP=no
EOF

# Create the veth network device
cat > /etc/systemd/network/mv-${1}-vs.netdev <<EOF
[NetDev]
Name=mv-${1}-v1
Kind=veth

[Peer]
Name=mv-${1}-v2
EOF

# plugin the first side of the veth-pair 
cat > /etc/systemd/network/mv-${1}-v1.network <<EOF
[Match]
Name=mv-${1}-v1

[Network]
Bridge=br-${1}
EOF

# plugin the other side of the veth-pair
cat > /etc/systemd/network/mv-${1}-v2.network <<EOF
[Match]
Name=mv-${1}-v2

[Network]
MACVLAN=mv-${1}
EOF

# Restart systemd-networkd
systemctl restart systemd-networkd.service

This script can be used by simply executing it and passing in the name of the bridge you want to integrate with. I saved the script as a file name mv-int-script.sh on my local system. As root run the following:

for interface in mgmt vlan vxlan storage flat; do
  bash ./mv-int-script.sh "$interface"
done

Verify Configuration

Once you've integrated with all of the bridges on your host machine, check the network list to ensure they're all online. Assuming everything is showing up as configured, you're good to go.

networkctl list
IDX LINK             TYPE               OPERATIONAL SETUP     
  1 lo               loopback           carrier     unmanaged 
  2 ens3             ether              routable    unmanaged 
  3 br-storage       ether              routable    unmanaged 
  4 eth12            ether              degraded    unmanaged 
  5 br-vlan-veth     ether              carrier     unmanaged 
  6 br-vlan          ether              routable    unmanaged 
  7 br-vxlan         ether              routable    unmanaged 
  8 br-container     ether              routable    configured
  9 dummy0           ether              degraded    configured
 10 mv-vxlan-v2      ether              degraded    configured
 11 mv-vxlan-v1      ether              degraded    configured
 12 mv-vlan-v2       ether              degraded    configured
 13 mv-vlan-v1       ether              degraded    configured
 14 mv-storage-v2    ether              degraded    configured
 15 mv-storage-v1    ether              degraded    configured
 16 mv-mgmt-v2       ether              degraded    configured
 17 mv-mgmt-v1       ether              degraded    configured
 18 mv-storage       ether              degraded    configured
 19 mv-vxlan         ether              degraded    configured
 20 mv-mgmt          ether              degraded    configured
 21 mv-vlan          ether              degraded    configured
 22 mv-container-v2  ether              degraded    configured
 23 mv-container-v1  ether              degraded    configured
 24 mv-container     ether              degraded    configured
 25 br-mgmt          ether              routable    unmanaged 

And then verify the network status

networkctl status
‚óŹ        State: routable
       Address: 10.1.167.190 on ens3
                172.29.244.100 on br-storage
                172.29.248.1 on br-vlan
                172.29.248.100 on br-vlan
                172.29.240.100 on br-vxlan
                10.0.3.2 on br-container
                172.29.236.100 on br-mgmt
                2001:4800:1ae1:18:f816:3eff:fe84:5510 on ens3
                fe80::f816:3eff:fe84:5510 on ens3
                fe80::f058:67ff:fe7d:aa82 on br-storage
                fe80::4c84:b2ff:fef7:1614 on eth12
                fe80::3821:d7ff:fe73:afd3 on br-vlan
                fe80::98ac:97ff:fecf:d91b on br-vxlan
                fe80::d8d9:8ff:fe6a:36a8 on br-container
                fe80::5c34:6cff:fe2f:d45d on dummy0
                fe80::d440:33ff:fea7:3f7d on mv-vxlan-v2
                fe80::fc0e:88ff:fe9a:d149 on mv-vxlan-v1
                fe80::a02e:cdff:fe82:a606 on mv-vlan-v2
                fe80::2891:39ff:feca:f070 on mv-vlan-v1
                fe80::30af:ccff:fe75:fa62 on mv-storage-v2
                fe80::5401:bff:fe9b:b9a0 on mv-storage-v1
                fe80::f08c:c0ff:fe7e:db5f on mv-mgmt-v2
                fe80::d041:f5ff:fee9:2377 on mv-mgmt-v1
                fe80::40b2:a8ff:fed3:a077 on mv-storage
                fe80::d05e:ffff:fe04:ff83 on mv-vxlan
                fe80::894:4aff:fefc:5fdf on mv-mgmt
                fe80::c7:e1ff:fec8:1d77 on mv-vlan
                fe80::7876:ecff:fee4:6819 on mv-container-v2
                fe80::20ed:44ff:feda:97d6 on mv-container-v1
                fe80::d0e2:e2ff:fed3:b283 on mv-container
                fe80::c004:9dff:fe1f:6269 on br-mgmt
                fe80::98dc:1fff:fe8a:d2b6 on brq2c0f1c0f-20
                fe80::f855:d2ff:fe47:9d33 on brq915f42bf-cf
       Gateway: 10.0.0.1 on ens3
                10.0.3.1 on br-container

Assuming all of the interfaces expected are present, have addresses, and are configured everything should be ready to receive container workloads.