Nova-Compute + LVM, an IOPS Love Story
The OpenStack Nova project has long had the ability to provision instances into logical volumes however in my travels I've found most deployments don't use it as the go-to instance storage. However recently I've been asked by a couple folks how it can be done with OpenStack-Ansible.
This post will will simply show how I would configure OpenStack-Ansible to deploy Nova-Compute using logical volumes for instances storage and a little bit about why this might be desirable.
The setup
This post assumes a suitable volume group is available on the host. For me, this will be vg01.
compute1:~# vgdisplay vg01
--- Volume group ---
VG Name vg01
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 19
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 6
Open LV 6
Max PV 0
Cur PV 1
Act PV 1
VG Size 1.82 TiB
PE Size 4.00 MiB
Total PE 476931
Alloc PE / Size 98306 / 384.01 GiB
Free PE / Size 378625 / 1.44 TiB
VG UUID B2zktP-Sw96-TUlJ-H2vx-0Vuf-cu3d-p3mpsj
The hardware overview I have running within this environment can be seen here.
Configuring the Cloud
Add a configuration override for nova into the user_variables.yml file.
nova_nova_conf_overrides:
ephemeral_storage_encryption:
enabled: True
libvirt:
images_type: lvm
images_volume_group: vg01
volume_clear: zero
disk_cachemodes: none
In this example I'm using the OpenStack-Ansible configuration template capabilities to inject additional config into the nova.conf. This set of overrides will enable encrypted ephemeral disks, set the instance image type to lvm, define the volume group to deploy into and how logical volume slices, define the cleaned method when an instance is destroyed and set the disk cache mode so that we're able to take advantage of native IO. More of the possible Nova configuration options can be seen here.
Deploying the configuration
-
If the deployment is a new one, simply run the playbooks normally once the override is in place.
-
If this is an existing deployment we can reconfigure nova accordingly using the playbooks limits and tags.
openstack-ansible os-nova-install.yml --tags nova-config --limit nova_compute
After the playbooks have completed the required configuration will be in place and the nova-compute service will have been restarted.
Making sure it all works
Provision an instance to the reconfigured compute node(s). Once the instance is active, you can log into the compute host and run lvs which will show logical volumes using an instance UUID as it's base name.
Use the lvdisplay command to inspect the online volumes.
compute1:~# lvdisplay /dev/vg01/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX*
--- Logical volume ---
LV Path /dev/vg01/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk
LV Name XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk
VG Name vg01
LV UUID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
LV Write Access read/write
LV Creation host, time compute1, 2017-12-15 00:58:51 -0600
LV Status available
# open 2
LV Size 128.00 GiB
Current LE 32768
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3
--- Logical volume ---
LV Path /dev/vg01/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk.eph0
LV Name XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk.eph0
VG Name vg01
LV UUID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
LV Write Access read/write
LV Creation host, time compute1, 2017-12-15 00:59:06 -0600
LV Status available
# open 2
LV Size 64.00 GiB
Current LE 16384
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:4
--- Logical volume ---
LV Path /dev/vg01/XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk.swap
LV Name XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX_disk.swap
VG Name vg01
LV UUID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
LV Write Access read/write
LV Creation host, time compute1, 2017-12-15 00:59:07 -0600
LV Status available
# open 2
LV Size 4.00 MiB
Current LE 1
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:6
As seen in the output, I have a swap disk, an ephemeral disk, and a root disk. Which corresponds to my running instance.
compute1:~# nova show XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX
+--------------------------------------+----------------------------------------------------------+
| Property | Value |
+--------------------------------------+----------------------------------------------------------+
| LOCAL_NET network | 172.17.24.105 |
| OS-DCF:diskConfig | AUTO |
| OS-EXT-AZ:availability_zone | nova |
| OS-EXT-SRV-ATTR:host | compute1 |
| OS-EXT-SRV-ATTR:hostname | rpc-test-1 |
| OS-EXT-SRV-ATTR:hypervisor_hostname | compute1.rk-home |
| OS-EXT-SRV-ATTR:instance_name | instance-0000000b |
| OS-EXT-SRV-ATTR:kernel_id | |
| OS-EXT-SRV-ATTR:launch_index | 0 |
| OS-EXT-SRV-ATTR:ramdisk_id | |
| OS-EXT-SRV-ATTR:reservation_id | r-cwvc02ta |
| OS-EXT-SRV-ATTR:root_device_name | /dev/sda |
| OS-EXT-SRV-ATTR:user_data | - |
| OS-EXT-STS:power_state | 1 |
| OS-EXT-STS:task_state | - |
| OS-EXT-STS:vm_state | active |
| OS-SRV-USG:launched_at | 2017-12-15T06:59:19.000000 |
| OS-SRV-USG:terminated_at | - |
| accessIPv4 | |
| accessIPv6 | |
| config_drive | |
| created | 2017-12-15T06:58:44Z |
| description | - |
| flavor:disk | 128 |
| flavor:ephemeral | 64 |
| flavor:extra_specs | {} |
| flavor:original_name | m1.medium |
| flavor:ram | 12288 |
| flavor:swap | 4 |
| flavor:vcpus | 8 |
| hostId | fd8e91a39068e6eef6f902c1f325ed56dbcbf492b97d08c4d469dcec |
| host_status | UP |
| id | XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX |
| image | Ubuntu 16.04 (YYYYYYYY-YYYY-YYYY-YYYY-YYYYYYYYYYYY) |
| key_name | cloudnull |
| locked | False |
| metadata | {} |
| name | rpc-test-1 |
| os-extended-volumes:volumes_attached | [] |
| progress | 0 |
| security_groups | default |
| status | ACTIVE |
| tags | [] |
| tenant_id | 7ae65c813bb049eaab926a87a569d299 |
| updated | 2017-12-15T06:59:19Z |
| user_id | 35354f3c389a475fb8a3517c7c260c97 |
+--------------------------------------+----------------------------------------------------------+
Why you might care
While this is not a revolutionary post or an amazing revelation, it does show how easy it is to update a cloud using OpenStack-Ansible and articulates some of the flexibility & maturity OpenStack provides. As for why you should care, running instances in logical volumes can improve performance in terms of both IOPS and Latency, is very easy to manage, and is damn stable.
Great Success!
I hope this shows how simple it can be to make seemingly complex changes to an OpenStack-Ansible powered cloud. If you've applied this solution to your environment, for test or production, let me know how it works and I hope you enjoy the native IO :).