Migrating a BTRFS Volume Between Devices

When using BTRFS volumes it's sometimes necessary to move things around. BTRFS has some great tools to assist operators to move volumes and this post will cover one available method.

What is this post about

While the reasons to move a device from one place to another are numerous, this post will focus on moving a BTRFS loopback file to a logical volume in an effort to better confine the containers and improve IOPS. The work for this post originated from the OpenStack-Ansible project where I was running containerized workloads in LXC containers utilizing the machinectl backend. When I first started the environment I accepted all of the defaults and simply ran my deployment. While everything worked, the machinectl backend placed the containers within a loopback filesystem mounted at /var/lib/machines. While this is a great feature from a usability point of view, it can leave deployments in a less than performant state in some circumstances. Given that I always want to squeeze every ounce of performance out of my environments I set out to move my containers on a live system from one device to another without having to redeploy my workloads.

Getting started

Because this post is focused on containers I will be touching on LXC and systemd-nspawn container types. The techniques discussed in this post are not limited to containers at all. The ability to move a BTRFS volume from one device to another is universally available.

Before moving workloads to a new volume the workloads SHOULD be stopped. All services accessing the original volume should be halted while the data is migrated. Some folks may argue that this is not a requirement, and while those folks are not wrong, halting the workloads will ensure processes do not take prolonged periods of time and will vastly reduce the potential for data corruption.

Stopping containers

If the container workloads are LXC run the following to stop all containers.

# lxc-ls -f --active | awk '/RUNNING/ {print $1}' | xargs -n 1 lxc-stop -n

If the containers are systemd-nspawn run the following to stop all containers.

# machinectl list | awk '/container/ {print $1}' | xargs -n 1 machinectl poweroff

Identify the loopback volume

For the purpose of this example the "machines.raw" volume will be migrated from a loopback device into a physical volume.

Run the following command to get the device path for a given loopback file.

# losetup -a 

/dev/loop0: [51713]:10059 (/var/lib/machines.raw)

Take note of the device path, /dev/loop0, which will be used later.

Identify the mount path

Take the output of the loopback file and use grep to get the mount path.

mount | grep '/var/lib/machines.raw' 

/var/lib/machines.raw on /var/lib/machines type btrfs (rw,noatime,nodiratime,compress=lzo,space_cache,commit=120,subvolid=5,subvol=/)

Take note of the mount path, /var/lib/machines, which will be used later.

Create a new logical volume

This post is migrating from a loopback file to a logical volume. If the environment has a physical device or a logical partition that will be used for the migration this part can be omitted.

The following command will create a 100-gigabyte volume named "machines00" using the volume group vg00.

# lvcreate -n machines00 -L 100G vg00

Replace the old device

This command will replace the old device with a new device. Remember, this example is moving a loopback device to a logical volume. The origin device is /dev/loop0 and the target device is /dev/vg00/machines00 and the mount path is /var/lib/machines.

Modify these entries according to the environment.

btrfs replace start /dev/loop0 /dev/vg00/machines00 /var/lib/machines -f -B

This command will force the migration to the new target and run the command in the foreground. If the origin volume is large, this command may take some time to complete. In some circumstances, it may be good to run this command using a terminal multiplexer (tmux, screen).

After the migration has completed, resize the volume using the mount path.

btrfs filesystem resize max /var/lib/machines

Starting containers

If the container workloads are LXC run the following to start all containers.

# lxc-ls -f --active | awk '/RUNNING/ {print $1}' | xargs -n 1 lxc-start -dn

If the containers are systemd-nspawn run the following to start all containers.

# machinectl list | awk '/container/ {print $1}' | xargs -n 1 machinectl start

Update the persistent mount

If the persistent mount is in /etc/fstab, update the mount line item accordingly.

# Persistent BTRFS mount
/dev/vg00/machines00 /var/lib/machines btrfs defaults,noatime,nodiratime,compress=lzo,commit=120,space_cache 0 0

If the presistent mount will be using the built-in system-mount capability, create a mount unit file. The mount unit file must have a name which is representative of the mount path. In the file name translate "/" to "-". For this example the file will be created here /etc/systemd/system/var-lib-machines.mount with the following content.

[Unit]
Description=Auto mount for /var/lib/machines

[Mount]
What=/dev/vg00/machines00
Where=/var/lib/machines
Type=btrfs
Options=defaults,noatime,nodiratime,compress=lzo,commit=120,space_cache

[Install]
WantedBy=multi-user.target

The What entry is the new device path, the Where is the mount path. For more information on systemd-mounts have a look at the following post.

With the mount unit file in place, reload the daemon-reload.

systemctl daemon-reload

Start the mount.

systemctl start var-lib-machines.mount

Validate the migration and cleanup

With the old device migrated and the new mount started the BTRFS volume will be online and active. To validate the new mount is active, check the mounts with the following command and the mount path.

# mount | grep '/var/lib/machines'

/dev/mapper/vg00-machines00 on /var/lib/machines type btrfs (rw,noatime,nodiratime,compress=lzo,space_cache,commit=120,subvolid=5,subvol=/)

Validate that the device and mount path are the expected values.

If running with a systemd-mount the following command could be used to validate the status.

# systemctl status var-lib-machines.mount

● var-lib-machines.mount - Auto mount for /var/lib/machines
   Loaded: loaded (/etc/systemd/system/var-lib-machines.mount; enabled; vendor preset: enabled)
   Active: active (mounted) since Thu 2018-05-31 14:43:08 CDT; 1 months 17 days ago
    Where: /var/lib/machines
     What: /dev/mapper/vg00-machines00
    Tasks: 0 (limit: 8192)
   Memory: 28.0K
      CPU: 14ms

Once everything has been validated and you're comfortable with the setup, the original loopback file can be removed.

rm /var/lib/machines.raw

That was easy

I hope this post is informative and helpful. If you've used these tools and have been successful with them, reach out on twitter and let me know. I'd love to hear more about your experiences.