In FIWARELab, we recently upgraded from Openstack Icehouse to a Kilo High Availability (HA) deployment. Our approach involved having two simultaneous active Openstack deployments so VMs and volumes could be migrated from one deployment to another with a minimal downtime. More specifically, VMs and volumes were snapshotted and transferred to the Kilo deployment as images where they could be recreated by their users. During this process, we found that, VMs originally created from of CentOS images did not boot properly – their network interface did not come up correctly and the VM was unable to fetch user-metadata.
The root of the issue lies in a standard configuration of CentOS: its device manager (udev) saves a mapping between MAC address and network for security reasons and ensures that the network interface only boots up on that specific MAC address. This mapping is stored in a file which is created when a VM boots for the first time. The file is located in /etc/udev/rules.d/70-persistent-ipoib.rules.
This caused problems when we created a VM from a CentOS snapshot on the new system, Openstack Neutron creates a new network port – consequently a new MAC address is generated – and associates this port with the VM. At boot time, the VM creates a new interface with this new MAC address which is usually down by default. As this interface is down, the VM is unroutable and hence unable to fetch user-metadata even though, from an Openstack perspective, there is a functioning network interface attached to the VM.
In order to solve this, the new system should offer the same MAC address used previously – before taking the snapshot – to the VM. Neutron provides support for this: creating ports with specified MAC addresses that can be attached to the VM using Nova. The following commands show how to create a port with a specified MAC address (and IP address) in Neutron and how to attach that port to the VM:
neutron port-create --fixed-ip subnet_id=<subnet>,ip_address=<ip-address> --mac-address <mac-address> --tenant-id <user-tenant-id>
nova interface-attach --port-id <port-id> <instance-id>
One alternative solution, in case the VM is accessible from the VNC console, is to remove the file mentioned above and reboot the VM. In this case a new file will be created “updating” the mapping of MAC addresses and interfaces so the VM gets the MAC address provided by Openstack. The downside of this approach is that it requires access to the user VM; which most – if not all – cloud operators are not willing to do, unless no options are left. An alternative approach is to mount the VM image using a tool such as mount-image-callback and modify the above file there.
This is a specific issue that arose when performing VM migration between two Openstack deployments. It could also arise in other contexts in which snapshots of CentOS VMs are shared, although in most such cases, the VM users would be able to modify the network configuration. The guest machine was not able to adapt to configuration changes provided by the cloud system which in turn broke the guest machine networking which highlights how the operation of the guest and the underlying platform can often still be closely coupled resulting in unexpected issues.