As you may know, I run quite a few nspawn containers which are responsible for running my entire home network, entertainment system, home automation, and a few other applications here & there. What you may not be aware of is the fact that I'm running everything on top of BTRFS and systemd-nspawn. Systemd-nspawn expects a volume mounted at
/var/lib/machines using BTRFS. While you can run nspawn without BTRFS, doing so would mean missing out on features pre-packaged into systemd-nspawn. While BTRFS, nspawn, and systemd can do amazing things, the server and services still require periodic maintenance and today was one such day.
I received an alert today that one of my container hosts was filling up; it alerted notifying me that I was under a certain threshold of available storage. After logging into the node and ensuring everything was on the up-and-up I decided to grow the BTRFS partition by ~50GiB giving the server more room to breathe while also fulfilling some nerdy curiosity.
Here's what BTRFS was reporting when I logged into the box.
# btrfs filesystem df -h /var/lib/machines/ Data, single: total=50.00GiB, used=38.52GiB System, single: total=4.00MiB, used=16.00KiB Metadata, single: total=776.00MiB, used=355.91MiB GlobalReserve, single: total=128.00MiB, used=0.00B # btrfs filesystem show /var/lib/machines/ Label: none uuid: ffa12758-857f-4416-957d-4cfd71f2e42a Total devices 1 FS bytes used 38.87GiB devid 1 size 50.00GiB used 39.78GiB path /dev/mapper/vg--machines
The container file system
/var/lib/machines was by no means full but it was getting close and I didn't want it full. So I decided to extend the logical volume housing my BTRFS partition and grow the underlying file system.
BTRFS in a Logical Volume
As mentioned before, I use logical volumes, like millions of other folks running linux servers, and while some may question the value of LVM and BTRFS, there's no right answer and I find myself in the camp of folks still believing in the power and flexibility of LVM, that is until qgroups are fully flushed out and I have some time to build a bit more trust with them.
A quick overview of the volume I'll be extending.
# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert machines vg -wi-ao---- 50.00g [...]
To extend my volume, as a traditional Linux system administrator, I would simply run
lvresize -r -L+50G /dev/vg/machines, drink a beer, and be done, however, being BTRFS is in use things are a little different. If you happen to run this command on a logical volume that is also formatted BTRFS the
lvresize command will fail when it detects the BTRFS volume (I did this).
fsadm: Filesystem "btrfs" on device "/dev/mapper/vg--machines" is not supported by this tool fsadm failed: 1
While the resize command failed, the logical volume detail shows the command partially worked by adding the extents to the volume. This means I need to use the
btrfs command to extend the volume instead of just assuming
lvresize will do it for me.
# lvdisplay /dev/vg/machines --- Logical volume --- LV Path /dev/vg/machines LV Name machines VG Name vg LV UUID ucVj74-t7TN-HV8H-pokI-BYkW-grBY-4ryTn7 LV Write Access read/write LV Creation host, time carterhouse, 2017-04-23 11:30:31 -0500 LV Status available # open 1 LV Size 100.00 GiB Current LE 25600 Segments 1 Allocation inherit Read ahead sectors auto - currently set to 256 Block device 252:2
In the future, when I need to do this elsewhere, the correct command for extending the volume is to run
lvresizeas I did before but to omit the
BTRFS, a balancing & extension act
lvresize command grew the logical volume but it didn't resize the underlying file system. This means the BTRFS partition is still only 50GiB even though the surface area of the partition is now larger.
This next step is Not Required but will make sure BTRFS is fully aware of where everything is supposed to exist. To ensure BTRFS is happy, I'm going to balance the file system which will redistribute data and metadata across all available devices, in my case this a logical volume. Normally, this would be used when adding a device to an existing BTRFS volume group or any time we want to ensure there's an equal spread of blocks across the disks but in this instance I'm running the balance command for piece of mind.
btrfs command, balance the file system. While this will work online, the command may take a while, and tax the CPU, be patient.
btrfs balance start -v /var/lib/machines
Once the balancing act has completed, which can take some time, the command will output what was done.
Dumping filters: flags 0x7, state 0x0, force is off DATA (flags 0x0): balancing METADATA (flags 0x0): balancing SYSTEM (flags 0x0): balancing Done, had to relocate 12 out of 12 chunks
Assuming everything is happy and healthy, resize the partition to the max allowed.
btrfs filesystem resize max /var/lib/machines
Verify that the BTRFS volume is as big as it's supposed to be
# btrfs filesystem show /var/lib/machines/ Label: none uuid: ffa12758-857f-4416-957d-4cfd71f2e42a Total devices 1 FS bytes used 38.86GiB devid 1 size 100.00GiB used 39.78GiB path /dev/mapper/vg--machines
Finally, check the consumption size
# btrfs filesystem df -h /var/lib/machines/ Data, single: total=100.00GiB, used=38.52GiB System, single: total=4.00MiB, used=16.00KiB Metadata, single: total=776.00MiB, used=355.91MiB GlobalReserve, single: total=128.00MiB, used=0.00B
Log out and time to go get that beer.
That's all folks
With those couple of commands, my logical volume was extended, my file system was balanced, curated, and I've grown the home for my containers. All of which was done while running local workloads without any service interruptions. That was easy!