{"id":6980,"date":"2014-12-09T13:08:14","date_gmt":"2014-12-09T11:08:14","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=6980"},"modified":"2014-12-09T13:08:14","modified_gmt":"2014-12-09T11:08:14","slug":"performance-analysis-of-post-copy-live-migration-in-openstack","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/performance-analysis-of-post-copy-live-migration-in-openstack\/","title":{"rendered":"Performance analysis of “post-copy” live migration in Openstack"},"content":{"rendered":"

Previously<\/a> we described how to \u00a0set up post-copy live migration<\/a> in OpenStack<\/a> Icehouse<\/a> (and it should not be a problem to set it up in the same way in Juno<\/a>). Naturally, \u00a0we were curious to see how it performs. In this blog post we focus on performance analysis of post-copy live migration in Openstack Icehouse using QEMU<\/a> \/ KVM<\/a> with libvirt<\/a>.<\/p>\n

Generally, when we consider virtual machine live migration the most important performance characteristics are migration time, VM down time and the amount of data transferred via network. The reliability and predictability of the migration process are also important and they are also discussed here. Regarding the migration time, we measure the duration of the live migration using time stamps recorded in nova-compute logs (\/var\/log\/nova\/nova-compute.log<\/em>) on source and destination hosts. We used the iftop<\/a> tool for measuring amount of transferred data and the ping<\/a> tool to determine the VM downtime; we pinged the VM public interface every 100ms (ping -i 0.1 <vm.float.ing.ip><\/em>) from a machine outside the network and track the packets lost during the migration to make the calculation.<\/p>\n

All experiments were done on our mid-range 3-node experimental OpenStack Icehouse deployment (1x controller and 2x compute) using wp3-postcopy branch of QEMU 2.1.5<\/a>. and wp3-postcopy branch of Libvirt 1.2.11<\/a>. All nodes are the same IBM x3550 m4<\/a> servers using 24 VCPUs, 192 GB RAM, 2.7 TB storage and 1Gb\/s ethernet interfaces. Note that live migration performance strongly depends on your hardware system setup (especially network throughput) and even though the results in in your environment may differ, this blog post should give you a sneak peak to the state-of-the-art live migration mechanism.<\/p>\n

The main advantage of post-copy live migration is that it is guaranteed to terminate (unlike pre-copy which can result in a neverending migration in some cases). However, it is inherently less robust and any network interruption can cause VM failure. For these reasons Libvirt introduces a hybrid approach to live migration which is actually a combination of more traditional pre-copy migration followed by a phase of post-copy migration. In the ideal case the combination of pre-copy and post-copy guarantees finite migration convergence while minimizing the duration of the network-failure sensitive post-copy phase. The current implementation performs one iteration of pre-copy and then switches to the post-copy mode. That means that VMs with low memory change rate are safely migrated during the pre-copy phase and there is almost no need to perform the post-copy phase which focuses on changes to memory which occur during the migration process. If you are interested in knowing a bit more about the difference between standard pre-copy<\/a> and post-copy<\/a> live migration mechanisms you can read our earlier blog post<\/a>.<\/p>\n

It should be noted we observed that the migration process was very reliable in all experiments (unlike some of the earlier we performed using older versions of QEMU and Libvirt).<\/p>\n

Hybrid live migration of unloaded instances<\/b><\/p>\n

First of all we focused on basic live migration of freshly spawned VMs running a cloud image of Ubuntu<\/a> 14.04 within different standard OpenStack flavors (from \u201csmall\u201d to \u201cextra large\u201d – see table 1).<\/p>\n\n\n\n\n\n
Flavor<\/td>\ns<\/td>\nm<\/td>\nl<\/td>\nxl<\/td>\n<\/tr>\n
VCPU (#)<\/td>\n1<\/td>\n2<\/td>\n4<\/td>\n8<\/td>\n<\/tr>\n
RAM (GB)<\/td>\n2<\/td>\n4<\/td>\n8<\/td>\n16<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table 1 – Parameters of the Openstack Flavors used<\/p>\n

On average we were able to migrate small flavored instances in approximately 5.5 seconds (385 MB transferred), medium in 7.2 s (435 MB), large in 8.7 s (530 MB) and x-large in 11.3 s (680 MB). We can clearly see that not the all available RAM in the VM is transferred but just the part of the memory that has already been allocated. The downtime in all cases was less than 0.3 seconds.<\/p>\n\n\n\n\n\n\n
Flavor<\/td>\ns<\/td>\nm<\/td>\nl<\/td>\nxl<\/td>\n<\/tr>\n
Migration time [s]<\/td>\n5.4<\/td>\n7<\/td>\n8.7<\/td>\n11.3<\/td>\n<\/tr>\n
Downtime [s]<\/td>\n0.2<\/td>\n0.2<\/td>\n0.2<\/td>\n0.3<\/td>\n<\/tr>\n
Transferred data [MB]<\/td>\n385<\/td>\n435<\/td>\n530<\/td>\n681<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table 2 – Unloaded VM hybrid live migration performance<\/p>\n

\"unloadedtime\"<\/a>
Figure 1 – Migration time and downtime – unloaded VMs<\/figcaption><\/figure>\n
\"unloadeddata\"<\/a>
Figure 2 – Data transfer – unloaded VMs<\/figcaption><\/figure>\n

Hybrid live migration of \u201cstressed\u201d instances<\/b><\/p>\n

In our earlier experiments<\/a> with pre-copy live migration we observed cases of unsuccessful live migration due to a high rate of change of RAM in a VM. (Remember, pre-copy live migration fails if the VM\u2019s RAM change rate is higher than source-to-destination network throughput). In these experiments, we used the stress tool to simulate intensive memory load. This tool runs a given number of threads that are periodically allocating and freeing a specified amount of memory. Our earlier experiments with stress of an x-large VM and pre-copy migration showed problems migrating instances with more than 100MB of stressed memory. We wanted to see how the new live migration approach worked in the same context \u00a0to objectively compare with our previous work.<\/p>\n

Unlike the pre-copy approach, the hybrid approach has an upper bound on the amount of data that will be transferred between source and destination: all used memory will be transferred during the pre-copy phase and all memory changed will be transferred at most once<\/i> (unlike pre-copy) during the post-copy phase. Thus, in the worst case, \u00a0the maximum amount of migrated data is given by 2 x RAM size and maximum migration time is 2 x RAM size \/ network throughput.<\/p>\n

Increasing the amount of stressed memory in increments of of 256MB results in a clear linear increase of transferred data (2 x stressed memory + resources allocated by the system) and migration time (transferred data \/ network throughput).<\/p>\n\n\n\n\n\n
Stressed memory [MB]<\/td>\n256<\/td>\n512<\/td>\n768<\/td>\n1024<\/td>\n1280<\/td>\n1536<\/td>\n1792<\/td>\n2048<\/td>\n<\/tr>\n
Migration time [s]<\/td>\n16.4<\/td>\n21.3<\/td>\n25.9<\/td>\n30.0<\/td>\n34.7<\/td>\n39.7<\/td>\n44.0<\/td>\n48.7<\/td>\n<\/tr>\n
Transferred data [MB]<\/td>\n1202<\/td>\n1722<\/td>\n2202<\/td>\n2649<\/td>\n3108<\/td>\n3696<\/td>\n4208<\/td>\n4713<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table 3 – Migration time and data transfer with increasing stressed memory (step of 256MB)<\/p>\n

\"migtime256\"
Figure 3 – Migration time with increasing stressed memory (steps of 256MB)<\/figcaption><\/figure>\n
\"data256\"<\/a>
Figure 4 – Data transfer with increasing stressed memory (steps of 256MB)<\/figcaption><\/figure>\n

Once we were satisfied that \u00a0live migration was reliable for smaller amounts of stressed memory, we increased the increment to 1024MB \/ thread and stressed up to 8 x 1024MB of the VM\u2019s RAM and again observed the same linear increase of these variables.<\/p>\n\n\n\n\n\n
Stressed memory [MB]<\/td>\n1024<\/td>\n2048<\/td>\n3072<\/td>\n4096<\/td>\n5120<\/td>\n6144<\/td>\n7168<\/td>\n8192<\/td>\n<\/tr>\n
Migration time [s]<\/td>\n30.7<\/td>\n49.5<\/td>\n69.0<\/td>\n86.1<\/td>\n101.4<\/td>\n120.1<\/td>\n138.0<\/td>\n157.2<\/td>\n<\/tr>\n
Transferred data [MB]<\/td>\n2725<\/td>\n4773<\/td>\n6728<\/td>\n8760<\/td>\n10500<\/td>\n12550<\/td>\n14600<\/td>\n16125<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

[Table 4 – Migration time and data transfer with increasing stressed memory (step of 1024MB)]<\/p>\n

\"migtime1024\"<\/a>
Figure 5 – Migration time with increasing stressed memory (steps of 1024MB)<\/figcaption><\/figure>\n
\"data1024\"
Figure 6 – Data transfer with increasing stressed memory (steps of 1024MB)<\/figcaption><\/figure>\n

As can be seen from the data transfers in figure 6, using the stress tool we could see that all stressed memory was transferred twice – once during pre-copy and once during post copy – resulting in a total data transfer of very close to twice the amount of stressed memory.<\/p>\n

We have observed several interesting observation regarding the downtime:<\/p>\n