{"id":7651,"date":"2015-04-09T16:06:24","date_gmt":"2015-04-09T14:06:24","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=7651"},"modified":"2015-05-07T15:06:05","modified_gmt":"2015-05-07T13:06:05","slug":"tunneled-hybrid-live-migration","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/tunneled-hybrid-live-migration\/","title":{"rendered":"Tunneled Hybrid Live Migration"},"content":{"rendered":"

In our previous blog posts we mostly focused on virtual machine live migration performance comparing pre-copy<\/a>, post-copy<\/a> and hybrid approaches in an Openstack<\/a> context rather than exploring other live migration features. Libvirt<\/a> together with the Qemu<\/a> hypervisor provides many migration configuration options. One of these options is a possibility to use tunneled live migration. Recently we found that the current libvirt tunneling implementation is not supported in post-copy migration. Consequently, In order to make the post-copy patch more production ready we decided to support the community and add support for post-copy tunneled live migration to libvirt on our own. This blog post describes the whole story of immersing ourselves into the open source community and hacking an established open source project since we believe this experience can be generalized.<\/p>\n

Since live migration is essentially a process of transferring complete virtual machine RAM, the constituent data stream can contain sensitive and confidential user information; consequently, it is necessary to separate this traffic from the other guest network traffic. In many Openstack environments physical network separation or VLAN<\/a> tagging is used to isolate different traffic types. In the cases where traffic separation is not possible libvirt\u2019s tunneling feature can be used instead. However since tunneling uses extra libvirtd-to-Qemu and Qemu-to-libvirtd data handling on both the source and the destination side it may perform worse than native live migration. Libvirt RPC protocol<\/a> used for tunneled LM provides, amongst the other, data encryption. Also, using tunneling, there is no need for any hypervisor specific network and firewall configuration since only one port per host is used no matter how many simultaneous parallel migrations are in progress. In contrast using the native direct guest-to-guest migration Qemu opens a new connection on a new port for each migration separately. So, there are some good reasons why the tunneled approach could be considered in preference to the native approach.<\/p>\n

\"Untitled<\/a>
Fig 1. – Native post-copy live migration code flow<\/figcaption><\/figure>\n

To enable tunneled live migration in an Openstack context just add the VIR_MIGRATE_TUNNELLED flag to your current live_migration_flags in the nova.conf file, restart the nova-compute service and you are all set for tunneled live migration in most cases. Unfortunately, if you want to use tunneling for post-copy live migration this will not work. The reason why tunneled post-copy live migration doesn\u2019t work is simple – the post-copy algorithm by its very nature requires bi-directional communication between the source and the destination virtual machine while libvirt currently supports only unidirectional tunnels (which are sufficient for pre-copy live migration). The channel from the source to the destination is used for transferring of VM\u2019s memory pages. The other direction, from the destination to the source is used by the post-copy algorithm for requesting particular missing memory pages. (If you don\u2019t understand why, you need to read up a bit more on post-copy live migration<\/a>.)<\/p>\n

\"Post-copy<\/a>
Fig 2. – Tunneled post-copy live migration code flow<\/figcaption><\/figure>\n

Unfortunately libvirt\u2019s post-copy patch currently doesn\u2019t come with code providing bi-directional tunneling; therefore there are two options: a) don\u2019t use tunneled post-copy live migration or b) get pulled into the libvirt\u2019s code and implement this feature on your own.<\/p>\n

Due to the fact we really like working with post-copy live migration and we would love to see it finally upstream we decided for option b) implement bi-directional tunneling into the libvirt project – sounds simple, right? Not quite!<\/p>\n

The libvirt migration tunnel (the connection between source and destination libvirt daemons) is represented by the \u2018virStream<\/a>\u2019 object from libvirt-stream module<\/a>. This provides the functionality to handle standard stream events such as \u201cstream readable\u201d \/ \u201cstream writable\u201d occurring on both sides of the tunnel. These stream structures support \u00a0general data transfers (also out of the live migration context) and consequently have a number of different possible configurations: to support bidirectional communications, it was necessary to modify the migration related streams to operate in non-blocking mode using the \u2018VIR_STREAM_NONBLOCK\u2019 flag.<\/p>\n

Once the stream is able to both send and receive data the corresponding event handlers providing communication between the libvirt daemon and Qemu need to be modified as well. The uni-directional implementation uses unix pipe<\/a>s to transfer migration traffic between Qemu and libvirt daemon as it is simple and reliable tool for interprocess communication. To support bi-directional communication it was necessary to replace unidirectional pipes with bidirectional socketpair<\/a>s. Libvirt module libvirt-event<\/a> allows to handle events occurring on the FDs in the similar fashion as in the previous case with the streams.<\/p>\n

Although the basic building blocks are now in place to support bidirectional communications, the logic that sits on top of this must be added. The pre-copy implementation in \u2018qemuMigrationIOFunc\u2019 function awaits the data from source Qemu and repeatedly passes those to the migration stream until the migration process is finished completely. Post-copy needs to be able also read the data from the migration stream and send them back to the source Qemu. To do so we replaced the original function with two callback functions (\u2018qemuMigrationEventOnFD\u2019 and \u2018qemuMigrationEventOnStream\u2019) handling standard data events in both directions. To avoid blocking we also added two data buffers that are shown in Fig. 3. The bi-directional communication mechanism implemented can be summed up as follows:<\/p>\n