Running local services in network namespaces with systemd
Given systemd is ubiquitous with modern day GNU/Linux it only makes sense that it be able to do just about everything under the sun; which strictly adheres to the UNIX philosophy 😂.
While the move to systemd has not always been a popular one, I too once hated it, it has matured and is providing much value. As sytemd has matured, it's created a lot of beautiful features and has exposed some powerful functionality, most of which is easy to consume and understand after spending time reading the docs.
The Unix philosophy emphasizes building simple, short, clear, modular, and extensible code that can be easily maintained and repurposed by developers other than its creators. The Unix philosophy favors composability as opposed to monolithic design.
In this post I'm covering one such function, running systemd units in a network namespace. I'll cover how to run haproxy in a network namespace and how it can be connected on the same Layer 2 network as my physical host via a macvlan.
This setup shows how it's possible to have an isolated yet seamless experience all using built-in capabilities. No iptables rules have been abused to make any of this work.
Setup helper service units
The first thing to do is to create a couple of helper service units. Helper units execute alongside regular services, and in this case, they'll be the glue that allows a given service to join a network namespace using the calling service unit's name as the namespace identifier.
Initiallization Helper
[Unit]
Description=Named network namespace %i
JoinsNamespaceOf=systemd-netns@%i.service
BindsTo=systemd-netns-access@%i.service
PartOf=%i.service
After=syslog.target network.target systemd-netns@%i.service
[Service]
Type=oneshot
RemainAfterExit=true
PrivateNetwork=true
# Start process
ExecStartPre=-/usr/bin/env ip netns delete %I
ExecStart=/usr/bin/env ip netns add %I
ExecStart=/usr/bin/env ip netns exec %I ip link set lo up
ExecStart=/usr/bin/env umount /var/run/netns/%I
ExecStart=/usr/bin/env mount --bind /proc/self/ns/net /var/run/netns/%I
# Stop process
ExecStop=/usr/bin/env ip netns delete %I
[Install]
WantedBy=multi-user.target
WantedBy=network-online.target
This unit prepares a network namespace which the calling service is bound to. This service unit is joining the namespace of itself and is bound to another yet another helper unit.
Access Helper
[Unit]
Description=Named network namespace %I
After=syslog.target network.target systemd-netns@%i.service
Before=%i.service
BindsTo=systemd-netns@%i.service
[Service]
Type=oneshot
RemainAfterExit=true
# Create system process
ExecStartPre=-/usr/bin/env ip link add mv-int link ${GATEWAY_DEVICE} type macvlan mode bridge
ExecStartPre=-/usr/bin/env ip link set mv-int up
ExecStartPre=-/usr/bin/env sysctl -w net.ipv4.ip_forward=1
# Pivot link
ExecStart=/usr/bin/env ip link add mv0 link mv-int type macvlan mode bridge
ExecStart=/usr/bin/env ip link set mv0 netns %i name mv0
# Configure link
ExecStart=-/usr/bin/env ip netns exec %i ip link set lo up
ExecStart=-/usr/bin/env ip netns exec %i ip link set dev mv0 up
ExecStart=-/usr/bin/env if [[ -e "/usr/local/bin/ns-%i" ]]; then bash /usr/local/bin/ns-%i start; fi
ExecStop=/usr/bin/env if [[ -e "/usr/local/bin/ns-%i" ]]; then bash /usr/local/bin/ns-%i stop; fi
[Install]
WantedBy=multi-user.target
WantedBy=network-online.target
This herlp unit prepares a macvlan type device and binds it to the defined ethernet device (in my example I used the bash variable ${GATEWAY_DEVICE}). With the macvlan setup, an interface is created and enabled within the service network namespace. Finally, a local script is executed should one exist.
Start up script
This startup script is optional and not required to make a service execute from within a network namespace. This script has been provided as an example on how given IP addresses (VIPs) can be set within a network namespace.
#!/usr/bin/env bash
function start {
/usr/bin/env ip netns exec haproxy sysctl -w net.ipv4.conf.mv0.forwarding=1
/usr/bin/env ip netns exec haproxy sysctl -w net.ipv4.conf.mv0.arp_notify=1
/usr/bin/env ip netns exec haproxy sysctl -w net.ipv4.conf.mv0.arp_announce=2
/usr/bin/env ip netns exec haproxy sysctl -w net.ipv4.conf.mv0.use_tempaddr=0
/usr/bin/env ip netns exec haproxy ip address add 172.16.26.1/22 dev mv0
/usr/bin/env ip netns exec haproxy ip address add 172.16.26.2/22 dev mv0
/usr/bin/env ip route add 172.16.26.1/32 dev mv-int metric 100 table local
/usr/bin/env ip route add 172.16.26.2/32 dev mv-int metric 100 table local
}
function stop {
/usr/bin/env ip route del 172.16.26.1/32 dev mv-int metric 100 table local
/usr/bin/env ip route del 172.16.26.2/32 dev mv-int metric 100 table local
}
case "$1" in
start)
start
;;
stop)
stop
;;
restart)
stop
start
;;
*)
echo "Usage: $0 {start|stop|restart}"
esac
The simple start-up script adds virtual IP addresses to the "haproxy" namespace which matches the haproxy service name. It also ensures physical host traffic is able to route back to the namespace as needed.
A route is created as a local-only route which is needed because macvlan type interfaces do not allow for traffic to hairpin. This route creates a path back to the service network namespace thereby circumventing the haripin limitation.
Helper unit recap
These helper service units, while simple, provide a robust capability. Having the execute applications in namespaces opens up so many beautiful possibilities, like giving users the ability to run multiple instances of a single service without running into port conflicts or isolating a load balancer on the same physical host an application is running on.
Making this work
The magic of running any service in a network namespace using the above helper service units is done through service drop-ins. Service drop-ins provide the ability to modify an existing service unit, like the one provided by a package maintainer, without having to own or modify the existing unit file.
Create a service drop-in
To run services from within a network namespace using the two previously created helper units a drop-in configuration file is created for an existing service. In this case, the example application is haproxy.
To create a service drop-in, create a directory using the service unit name like so,
/etc/systemd/system/haproxy.service.d. Then create a configuration file,/etc/systemd/system/haproxy.service.d/haproxy.conf, for the drop-in content.
[Unit]
BindsTo = systemd-netns@haproxy.service
JoinsNamespaceOf = systemd-netns@haproxy.service
After = systemd-netns@haproxy.service
[Service]
CPUAccounting = true
BlockIOAccounting = true
MemoryAccounting = true
TasksAccounting = true
PrivateNetwork = true
PrivateTmp = true
Slice = haproxy.slice
The drop-in created in this example binds to and joins the namespace of the systemd-netns service unit. It also ensures the service is started after the network namespace is available, adds additional reporting around the functionality of the service, instruct the service to execute within a private namespace with a tmp, and executes from within a specific slice (cgroup).
Drop-in Overview
The net effect of this drop-in is a fully functional haproxy service wholly isolated in a network namespace though available via virtual IP addresses.
Using all of this in practice
The best part about this entire setup is the fact an operator can control a given service without having to worry about anything special going on under the hood.
- To show the status on the haproxy service that happens to be running in a network namespace.
root@utility:~# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/haproxy.service.d
└─haproxy.conf
Active: active (running) since Tue 2019-04-02 21:10:41 UTC; 2s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
file:/var/run/netns/haproxy
Process: 31015 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 31025 (haproxy)
Tasks: 2 (limit: 8192)
Memory: 8.7M
CPU: 22ms
CGroup: /haproxy.slice/haproxy.service
├─31025 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
└─31029 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
Apr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:288] : backend 'neutron_server-back' : 'option httplog' directive iApr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:315] : backend 'nova_api_metadata-back' : 'option httplog' directivApr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:348] : backend 'nova_api_os_compute-back' : 'option httplog' directApr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:375] : backend 'nova_api_placement-back' : 'option httplog' directiApr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:412] : backend 'nova_console-back' : 'option httplog' directive is Apr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:440] : backend 'rabbitmq_mgmt-back' : 'option httplog' directive isApr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:464] : backend 'repo_all-back' : 'option httplog' directive is ignoApr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:488] : backend 'repo_git-back' : 'option tcplog' directive is ignorApr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:521] : backend 'swift_proxy-back' : 'option httplog' directive is iApr 02 21:10:41 utility systemd[1]: Started HAProxy Load Balancer.
- To stop a service that happens to be running within a network namespace.
root@utility:~# systemctl stop haproxy
root@utility:~# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/haproxy.service.d
└─haproxy.conf
Active: failed (Result: exit-code) since Tue 2019-04-02 21:11:27 UTC; 7s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
file:/var/run/netns/haproxy
Process: 31025 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, status=143)
Process: 31015 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 31025 (code=exited, status=143)
CPU: 49ms
Apr 02 21:10:41 utility haproxy[31025]: [WARNING] 091/211041 (31025) : parsing [/etc/haproxy/haproxy.cfg:521] : backend 'swift_proxy-back' : 'option httplog' directive is iApr 02 21:10:41 utility systemd[1]: Started HAProxy Load Balancer.
Apr 02 21:11:27 utility systemd[1]: Stopping HAProxy Load Balancer...
Apr 02 21:11:27 utility haproxy[31025]: [WARNING] 091/211041 (31025) : Exiting Master process...
Apr 02 21:11:27 utility haproxy[31025]: [ALERT] 091/211041 (31025) : Current worker 31029 exited with code 143
Apr 02 21:11:27 utility haproxy[31025]: [WARNING] 091/211041 (31025) : All workers exited. Exiting... (143)
Apr 02 21:11:27 utility systemd[1]: haproxy.service: Main process exited, code=exited, status=143/n/a
Apr 02 21:11:27 utility systemd[1]: haproxy.service: Failed with result 'exit-code'.
Apr 02 21:11:27 utility systemd[1]: Stopped HAProxy Load Balancer.
Apr 02 21:11:27 utility systemd[1]: haproxy.service: Consumed 49ms CPU time
- To start a service that happens to be running within a network namespace.
root@utility:~# systemctl start haproxy
root@utility:~# systemctl status haproxy
● haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Drop-In: /etc/systemd/system/haproxy.service.d
└─haproxy.conf
Active: active (running) since Tue 2019-04-02 21:12:23 UTC; 1s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
file:/var/run/netns/haproxy
Process: 31706 ExecStartPre=/usr/sbin/haproxy -f $CONFIG -c -q $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 31708 (haproxy)
Tasks: 2 (limit: 8192)
Memory: 8.2M
CPU: 28ms
CGroup: /haproxy.slice/haproxy.service
├─31708 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
└─31718 /usr/sbin/haproxy -Ws -f /etc/haproxy/haproxy.cfg -p /run/haproxy.pid
Apr 02 21:12:23 utility haproxy[31708]: [WARNING] 091/211223 (31708) : parsing [/etc/haproxy/haproxy.cfg:288] : backend 'neutron_server-back' : 'option httplog' directive iApr 02 21:12:23 utility haproxy[31708]: [WARNING] 091/211223 (31708) : parsing [/etc/haproxy/haproxy.cfg:315] : backend 'nova_api_metadata-back' : 'option httplog' directivApr 02 21:12:23 utility haproxy[31708]: [WARNING] 091/211223 (31708) : parsing [/etc/haproxy/haproxy.cfg:348] : backend 'nova_api_os_compute-back' : 'option httplog' directApr 02 21:12:23 utility haproxy[31708]: [WARNING] 091/211223 (31708) : parsing [/etc/haproxy/haproxy.cfg:375] : backend 'nova_api_placement-back' : 'option httplog' directiApr 02 21:12:23 utility haproxy[31708]: [WARNING] 091/211223 (31708) : parsing [/etc/haproxy/haproxy.cfg:412] : backend 'nova_console-back' : 'option httplog' directive is Apr 02 21:12:23 utility haproxy[31708]: [WARNING] 091/211223 (31708) : parsing [/etc/haproxy/haproxy.cfg:440] : backend 'rabbitmq_mgmt-back' : 'option httplog' directive isApr 02 21:12:23 utility haproxy[31708]: [WARNING] 091/211223 (31708) : parsing [/etc/haproxy/haproxy.cfg:464] : backend 'repo_all-back' : 'option httplog' directive is ignoApr 02 21:12:23 utility haproxy[31708]: [WARNING] 091/211223 (31708) : parsing [/etc/haproxy/haproxy.cfg:488] : backend 'repo_git-back' : 'option tcplog' directive is ignorApr 02 21:12:23 utility haproxy[31708]: [WARNING] 091/211223 (31708) : parsing [/etc/haproxy/haproxy.cfg:521] : backend 'swift_proxy-back' : 'option httplog' directive is iApr 02 21:12:23 utility systemd[1]: Started HAProxy Load Balancer.
As you can see, there's no funny business. From the vantage of an operator or administrator, services executed in a network namespace are identical to everything else running on a given system.
Validating the network namespace is functional
Validating network namespaces are working as expected is simple. Attach to the network namespace and ensure the vips and expected ports are all online.
- List your network namespaces.
root@utility:~# ip netns
haproxy (id: 2)
- List the network devices from within your service network namespace.
root@utility:~# ip netns exec haproxy ip a l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
122: mv0@if5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether ae:7a:de:55:9f:92 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.16.26.1/22 scope global mv0
valid_lft forever preferred_lft forever
inet 172.16.26.2/22 scope global secondary mv0
valid_lft forever preferred_lft forever
inet6 fe80::ac7a:deff:fe55:9f92/64 scope link
valid_lft forever preferred_lft forever
- Check the expected ports are all online.
root@utility:~# ip netns exec haproxy ss -ntlp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 172.16.26.2:8004 0.0.0.0:* users:(("haproxy",pid=31718,fd=12))
LISTEN 0 128 172.16.26.1:8004 0.0.0.0:* users:(("haproxy",pid=31718,fd=11))
LISTEN 0 128 172.16.26.2:8774 0.0.0.0:* users:(("haproxy",pid=31718,fd=25))
LISTEN 0 128 172.16.26.1:8774 0.0.0.0:* users:(("haproxy",pid=31718,fd=24))
LISTEN 0 128 172.16.26.2:8775 0.0.0.0:* users:(("haproxy",pid=31718,fd=23))
LISTEN 0 128 172.16.26.2:5000 0.0.0.0:* users:(("haproxy",pid=31718,fd=20))
LISTEN 0 128 172.16.26.1:5000 0.0.0.0:* users:(("haproxy",pid=31718,fd=19))
LISTEN 0 128 172.16.26.2:8776 0.0.0.0:* users:(("haproxy",pid=31718,fd=7))
LISTEN 0 128 172.16.26.1:8776 0.0.0.0:* users:(("haproxy",pid=31718,fd=5))
LISTEN 0 128 172.16.26.2:9418 0.0.0.0:* users:(("haproxy",pid=31718,fd=31))
LISTEN 0 128 172.16.26.2:3306 0.0.0.0:* users:(("haproxy",pid=31718,fd=8))
LISTEN 0 128 172.16.26.2:8780 0.0.0.0:* users:(("haproxy",pid=31718,fd=26))
LISTEN 0 128 172.16.26.2:9292 0.0.0.0:* users:(("haproxy",pid=31718,fd=10))
LISTEN 0 128 172.16.26.1:9292 0.0.0.0:* users:(("haproxy",pid=31718,fd=9))
LISTEN 0 128 172.16.26.2:8080 0.0.0.0:* users:(("haproxy",pid=31718,fd=33))
LISTEN 0 128 172.16.26.1:8080 0.0.0.0:* users:(("haproxy",pid=31718,fd=32))
LISTEN 0 128 172.16.26.2:80 0.0.0.0:* users:(("haproxy",pid=31718,fd=17))
LISTEN 0 128 172.16.26.1:80 0.0.0.0:* users:(("haproxy",pid=31718,fd=15))
LISTEN 0 128 172.16.26.2:8181 0.0.0.0:* users:(("haproxy",pid=31718,fd=30))
LISTEN 0 128 172.16.26.2:15672 0.0.0.0:* users:(("haproxy",pid=31718,fd=29))
LISTEN 0 128 172.16.26.2:443 0.0.0.0:* users:(("haproxy",pid=31718,fd=18))
LISTEN 0 128 172.16.26.1:443 0.0.0.0:* users:(("haproxy",pid=31718,fd=16))
LISTEN 0 128 172.16.26.2:9696 0.0.0.0:* users:(("haproxy",pid=31718,fd=22))
LISTEN 0 128 172.16.26.1:9696 0.0.0.0:* users:(("haproxy",pid=31718,fd=21))
LISTEN 0 128 172.16.26.2:8000 0.0.0.0:* users:(("haproxy",pid=31718,fd=14))
LISTEN 0 128 172.16.26.1:8000 0.0.0.0:* users:(("haproxy",pid=31718,fd=13))
LISTEN 0 128 172.16.26.2:6082 0.0.0.0:* users:(("haproxy",pid=31718,fd=28))
LISTEN 0 128 172.16.26.1:6082 0.0.0.0:* users:(("haproxy",pid=31718,fd=27))
Connectivity to the network namespace can be validated through the connected macvlan device on the physical host machine.
root@utility:~# ip a l mv-int
73: mv-int@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 86:cf:ad:34:7f:4c brd ff:ff:ff:ff:ff:ff
We can then ping the virtual IP addresses from the physical host to ensure
the local route is working as expected.
root@utility:~# ping -I mv-int 172.16.26.2 -c 3
ping: Warning: source address might be selected on device other than mv-int.
PING 172.16.26.2 (172.16.26.2) from 172.16.27.217 mv-int: 56(84) bytes of data.
64 bytes from 172.16.26.2: icmp_seq=1 ttl=64 time=0.025 ms
64 bytes from 172.16.26.2: icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from 172.16.26.2: icmp_seq=3 ttl=64 time=0.042 ms
Finally we can cURL one of the APIs I have running on my local haproxy environment.
root@utility:~# curl -s 172.16.26.2:5000 | python -m json.tool
{
"versions": {
"values": [
{
"id": "v3.12",
"links": [
{
"href": "http://172.16.26.2:5000/v3/",
"rel": "self"
}
],
"media-types": [
{
"base": "application/json",
"type": "application/vnd.openstack.identity-v3+json"
}
],
"status": "stable",
"updated": "2019-01-22T00:00:00Z"
}
]
}
}
That was easy!
Given this is the longest post ever covering how two files can unlock the ability to run services in network namespaces using macvlan interfaces for layer 2 connectivity and how one drop-in config file can extend the functionality of any services already installed on a local system, I think I'll wrap things up.
That's all for now
That's all folks. I hope this post was informative, helps shine a light on some of the built-in capabilities systemd comes with out of the box, and illustrates how some of the more obscure capabiltiies packed within systemd can be used to solve problems in unique and innovative ways.
For those who might want to do something similar, or are curious, I've created a git repository with the various service units within them and my haproxy drop-in configuration file. The repository can be seen here.