/ cloud

Rolling Restart using OpenStack-Ansible

cloud | osa | openstack-ansible | openstack | bash | restarts

Sometimes it's useful to do a rolling restart of OpenStack services throughout the cloud. In OpenStack-Ansible we've the means to make this easy, you just need a little BASH to tie it all together. In this little aside, I'm restarting the entire cloud using the OpenStack-Ansible inventory.

Listing possible services

To get a list of all of the services I could simply take all of the install playbooks as a list using something like this:

ls -1 *.yml | awk -F'-' '/os-/ {print $2}' | uniq

However, I don't want to simply take the order in which the files are found as a means to restart the environment. I want to run this operation in a specific order to lessen the impact seen within the environment.

Bashing the array

To make this a more controlled restart I will be building a simple bash variable with the service name. I will then iterate over the list and restart all discovered processes on the target nodes.

RESTART_ORDER=$(awk '/include:/ {print $3}' setup-openstack.yml | uniq | awk -F'-' '{print $2}')

For completeness, I added all services found in the openstack-setup.yml playbook, however, this list would likely be shorter and could have other, non-standard, services. Customize this list to suit your needs.

Iterating my way to success

Tthe loop I used to iterate over the list will restart all discovered services on the target nodes within the /etc/init and /etc/init.d directories. After the services are restarted the command will wait 3 seconds before continuing the rest of the loop. The anisble command will also limit the forks to 2 which will ensure we're not restarting everything at the same time within a given group. Customize the sleep time and forks to meet your needs and level of comfort.

for item in ${RESTART_ORDER}; do
  ansible -m shell -a "(systemctl list-unit-files --state=enabled --type=service || ls -1 /etc/init/ /etc/init.d/) | grep -e \"${item}\" -e apache -e nginx  | awk -F'.' '{print \$1}' | xargs -I{} -n 1 service {} restart && sleep 3" "${service_group}_all" --forks 2
done

With this simple loop, we're able to do a rolling restart across an entire OpenStack cloud without having to run a whole host of ad-hoc commands all using the OpenStack-Ansible inventory.

That was easy!

Kevin Carter

Kevin Carter

I'm me! Developer, Operator, Cloud Builder, Lover of Open Source, and hater of Nonsense (I know they're all related, leave me alone).

Read More