{"id":6063,"date":"2014-09-18T18:17:57","date_gmt":"2014-09-18T16:17:57","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=6063"},"modified":"2014-09-29T11:07:50","modified_gmt":"2014-09-29T09:07:50","slug":"an-analysis-of-the-performance-of-live-migration-in-openstack","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/an-analysis-of-the-performance-of-live-migration-in-openstack\/","title":{"rendered":"An analysis of the performance of live migration in Openstack"},"content":{"rendered":"

We continue our recent work regarding an analysis of the performance of live migration in Openstack<\/a> Icehouse. Our previous results<\/a> focused on block live migration in Openstack, without shared storage configured between computing nodes. In this post we focus on the performance of live migration in the system with a shared file system configured, compare it with block live migration and try to determine scenarios more suitable for each approach. [Note that in the above, when live migration implies shared storage based live migration; block live migration is explicit as in the Openstack literature on this topic].<\/i><\/p>\n

[Also note that all VMs in following experiments use 5GB of disk space]<\/em><\/p>\n

All system configuration and testing aspects (except amended flavors) remain the same as in previous block live migration tests (see here<\/a> for more details). The only infrastructure change in our small 3 node configuration is the shared file system setup. We used an NFS<\/a> server running on the controller node to provide its ..\/nova\/instances folder as a mount point for the compute nodes.<\/p>\n

Unloaded case<\/strong><\/p>\n

Let\u2019s start with the time taken by live migrations of unloaded VMs. In chart 1 we see a small increase of migration time with larger VM flavors, which is not so surprising. More interestingly the down time stays almost constant (< 1 s) and doesn\u2019t grow with the migration time unlike the case of block live migration.<\/p>\n\n\n\n\n\n
<\/td>\nt<\/strong><\/td>\ns<\/strong><\/td>\nm<\/strong><\/td>\nl<\/strong><\/td>\nxl<\/strong><\/td>\n<\/tr>\n
Migration time (unloaded)<\/strong><\/td>\n4.9<\/td>\n5.6<\/td>\n7.0<\/td>\n9.6<\/td>\n15.3<\/td>\n<\/tr>\n
Down time (unloaded)<\/strong><\/td>\n0.7<\/td>\n0.7<\/td>\n0.8<\/td>\n0.8<\/td>\n0.9<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table 1 – Live migration duration & down time<\/p>\n

\"mig_down_time_block\"<\/a>
Chart 1 – Live migration duration & down time<\/figcaption><\/figure>\n

 <\/p>\n

In all cases with shared storage, live migration of unloaded VMs exhibits significantly lower migration time and downtime than block live migration. More precisely, the NFS live migration time is on average only 60% of the block live migration time (see chart 2). VM down time is on average 17% of the down time using block migration (see chart 3) and is less than 1 s in our experiments.<\/p>\n\n\n\n\n\n
<\/td>\nt<\/strong><\/td>\ns<\/strong><\/td>\nm<\/strong><\/td>\nl<\/strong><\/td>\nxl<\/strong><\/td>\n<\/tr>\n
Live migration (unloaded)<\/strong><\/td>\n4.9<\/td>\n5.6<\/td>\n7.0<\/td>\n9.6<\/td>\n15.3<\/td>\n<\/tr>\n
Block live migration (unloaded)<\/strong><\/td>\n9.3<\/td>\n10.5<\/td>\n11.7<\/td>\n14.6<\/td>\n21.8<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table 2 – Migration time (unloaded) – live migration vs. block live migration<\/p>\n

\"\"<\/a>
Chart 2 – Migration time (unloaded) – live migration vs. block live migration<\/figcaption><\/figure>\n

 <\/p>\n\n\n\n\n\n
<\/td>\nt<\/strong><\/td>\ns<\/strong><\/td>\nm<\/strong><\/td>\nl<\/strong><\/td>\nxl<\/strong><\/td>\n<\/tr>\n
Live migration (unloaded)<\/strong><\/td>\n0.7<\/td>\n0.7<\/td>\n0.8<\/td>\n0.8<\/td>\n0.9<\/td>\n<\/tr>\n
Block live migration (unloaded)<\/strong><\/td>\n2.4<\/td>\n3.2<\/td>\n4.4<\/td>\n7.0<\/td>\n21.5<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table 3 – Down time (unloaded) – live migration vs. block live migration<\/p>\n

\"\"<\/a>
Chart 3 – Down time – live migration vs. block live migration (unloaded)<\/figcaption><\/figure>\n

 <\/p>\n

In the unloaded VM scenario, the amount of data transferred over the network increases with the VM size, however even these values are significantly smaller than in the block live migration case: Network traffic is 11 – 30% lower in the storage based migration case, than the block migration case.<\/p>\n\n\n\n\n\n
<\/td>\nt<\/strong><\/td>\ns<\/strong><\/td>\nm<\/strong><\/td>\nl<\/strong><\/td>\nxl<\/strong><\/td>\n<\/tr>\n
Live migration (unloaded)<\/strong><\/td>\n175<\/td>\n190<\/td>\n288<\/td>\n315<\/td>\n480<\/td>\n<\/tr>\n
Block live migration (unloaded)<\/strong><\/td>\n239<\/td>\n271<\/td>\n324<\/td>\n404<\/td>\n669<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
\"\"<\/a>
Chart 4 – Network data transfer (unloaded) – live migration vs. block live migration<\/figcaption><\/figure>\n

 <\/p>\n

Loaded case<\/b><\/p>\n

Behaviour of live migration for loaded machines was a little more complex.<\/p>\n

Using the block live migration we successfully migrated VMs across all flavors even when they were quite heavily loaded – in our tests, we focused on memory load (as described here<\/a>) and we achieved successful migration even when they were approximately 75% loaded. However, shared storage based live migration was more sensitive to memory load. \u00a0In fact, we couldn\u2019t migrate even a tiny flavor VM with a relatively small memory load – 100 MB of stressed memory (out of total 512 MB). Migration was initiated but didn\u2019t end until the load was terminated. Consequently, we can\u2019t present results which are directly comparable to the block live migration for this loaded case.<\/p>\n

To have a better understanding why we face this problem of non deterministic migration time we need to dive a bit deeper into the migration mechanism. In our configuration of Openstack with QEMU\/KVM and libvirt, the live migration uses pre-copy memory<\/a> approach. This consists of following steps:<\/p>\n

    \n
  1. Copy source VM memory to destination. This operation takes some time while VM is still running (and some memory pages are changing) at the source.<\/li>\n
  2. Repeat step 1 (copy changed memory pages) until amount of dirty memory is small enough and can be transferred in very short VM downtime<\/li>\n
  3. Stop VM at source and copy the rest of dirty memory<\/li>\n
  4. Start VM at destination<\/li>\n<\/ol>\n

    The problem arises at the point 2: if the VM memory is dirtied faster than it can be transferred to destination then this step will never terminate.. Therefore, whether a migration is a success or failure depends on the available network capacity \u00a0and the current VM memory load.<\/p>\n

    To explore this further, we considered VMs with different memory load and examined how it affects migration time and the amount of transferred data. Up to a stress level of 96 MB all migrations were \u00a0successful with relatively small constant downtime (<1 s) and short migration duration (mostly < 20 s). However, beyond this point the VM got stuck in a migration state and doesn\u2019t stop migrating (until it is either killed or the memory load reduced). The real measured throughput in our network was approximately 110 MB\/s, that\u2019s why we could migrate VM with lower memory load than this value and we failed with VMs, where the stress rate of memory exceeded this threshold.\u00a0<\/strong><\/p>\n\n\n\n\n
    Stressed memory [MB]<\/strong><\/td>\n0<\/strong><\/td>\n16<\/strong><\/td>\n32<\/strong><\/td>\n48<\/strong><\/td>\n64<\/strong><\/td>\n80<\/strong><\/td>\n96<\/strong><\/td>\n<\/tr>\n
    Migration time [s]<\/strong><\/td>\n15.0<\/td>\n15.3<\/td>\n16.3<\/td>\n17.3<\/td>\n18.5<\/td>\n20.8<\/td>\n20.8<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Table 5 – Migration time with different levels of stress<\/p>\n

    \"mig_time_increasing
    Chart 5 – Migration time with different levels of stress<\/figcaption><\/figure>\n

     <\/p>\n\n\n\n\n
    Stressed memory [MB]<\/strong><\/td>\n0<\/strong><\/td>\n16<\/strong><\/td>\n32<\/strong><\/td>\n48<\/strong><\/td>\n64<\/strong><\/td>\n80<\/strong><\/td>\n96<\/strong><\/td>\n<\/tr>\n
    Data transferred [MB]<\/strong><\/td>\n472<\/td>\n510<\/td>\n541<\/td>\n573<\/td>\n635<\/td>\n770<\/td>\n943<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

    Table 6 – Data transfer with different levels of stress<\/p>\n

    \"data_transfered_increasing_stress\"
    Chart 6 – Data transfer with different levels of stress<\/figcaption><\/figure>\n

     <\/p>\n

    Take aways<\/b><\/p>\n

    Openstack live migration is fast technique how to transfer running VM from one host to another with a small downtime. But since its duration is non deterministic and there is no guarantee that migration process finishes successfully, care must be taken when employing this approach and your particular network configuration and load as well as the VM activity must be considered.<\/p>\n

    Key take aways from our experiments are:<\/p>\n