Rolling Restart using OpenStack-Ansible
Sometimes it's useful to do a rolling restart of OpenStack services throughout the cloud. In OpenStack-Ansible we've the means to make this easy, you just need a little BASH to tie it all together. In this little aside, I'm restarting the entire cloud using the OpenStack-Ansible inventory.
Listing possible services
To get a list of all of the services I could simply take all of the install playbooks as a list using something like this:
ls -1 *.yml | awk -F'-' '/os-/ {print $2}' | uniq
However, I don't want to simply take the order in which the files are found as a means to restart the environment. I want to run this operation in a specific order to lessen the impact seen within the environment.
Bashing the array
To make this a more controlled restart I will be building a simple bash variable with the service name. I will then iterate over the list and restart all discovered processes on the target nodes.
RESTART_ORDER=$(awk '/include:/ {print $3}' setup-openstack.yml | uniq | awk -F'-' '{print $2}')
For completeness, I added all services found in the openstack-setup.yml playbook, however, this list would likely be shorter and could have other, non-standard, services. Customize this list to suit your needs.
Iterating my way to success
Tthe loop I used to iterate over the list will restart all discovered services on the target nodes within the /etc/init and /etc/init.d directories. After the services are restarted the command will wait 3 seconds before continuing the rest of the loop. The anisble command will also limit the forks to 2 which will ensure we're not restarting everything at the same time within a given group. Customize the sleep time and forks to meet your needs and level of comfort.
for item in ${RESTART_ORDER}; do
ansible -m shell -a "(systemctl list-unit-files --state=enabled --type=service || ls -1 /etc/init/ /etc/init.d/) | grep -e \"${item}\" -e apache -e nginx | awk -F'.' '{print \$1}' | xargs -I{} -n 1 service {} restart && sleep 3" "${service_group}_all" --forks 2
done
With this simple loop, we're able to do a rolling restart across an entire OpenStack cloud without having to run a whole host of ad-hoc commands all using the OpenStack-Ansible inventory.