{"id":6204,"date":"2014-10-14T14:12:48","date_gmt":"2014-10-14T12:12:48","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=6204"},"modified":"2014-10-14T14:12:48","modified_gmt":"2014-10-14T12:12:48","slug":"the-impact-of-ephemeral-vm-disk-usage-on-the-performance-of-live-migration-in-openstack","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/the-impact-of-ephemeral-vm-disk-usage-on-the-performance-of-live-migration-in-openstack\/","title":{"rendered":"The impact of ephemeral VM disk usage on the performance of Live Migration in Openstack"},"content":{"rendered":"

In our previous work we presented the performance of live migration in Openstack<\/a> Icehouse<\/a> using various types of VM flavors, its memory load and also examined how it performs in network and CPU loaded environment (see our previous posts –performance of live migration<\/a>, performance of block live migration<\/a>, performance of both under varying cpu and network load<\/a>). One factor which was not considered in our earlier work is the impact of VM ephemeral disk size on the performance of the live migration. That is the focus of this post.<\/p>\n

In the following experiments, then,we simply increased disk usage by consuming more of the disk capacity and observed how it affects both block live migration (BLM – disk migrated via the network) and live migration (LM – using a \u00a0shared file system between computing nodes).<\/p>\n

As described in earlier post<\/a>s, our Openstack setup uses 1 controller and 2 compute nodes connected with 1G Ethernet LAN network.<\/p>\n

Our first observation was that VM disk capacity itself (as defined by the flavor) doesn\u2019t significantly affect live migration performance. Rather, what impacts the performance is the amount of data (somehow) allocated on the filesystem. Unallocated disk space is not migrated at all, but once the VM makes any filesystem allocation of any bytes of its disk, these bytes will be migrated irrespective of whether the original file still exists or not. Clearly, this means that freshly spawned VMs with different disk capacities will migrate in the almost same time, because they allocate approximately the same amount of data on the filesystem – e.g. freshly installed Ubuntu Server 14.04 LTS<\/a> uses approximately 800 MB of disk space. Since we haven\u2019t generated any additional data except the OS in any of our previous tests with 5 GB ephemeral disks, those results can be generalized for freshly spawned VMs with larger disks as well.<\/p>\n

The following paragraphs describe performance of LM, BLM and their comparison with increasing disk usage. Migration time values were obtained from nova-all.log file. Downtime is the time period during that migrated VM doesn\u2019t reply to ping<\/a> echo requests. Transferred data is amount of data transferred via network from source to destination host (measured by iftop<\/a>).<\/p>\n

Block Live Migration<\/b><\/p>\n

In the case of BLM both the migration time and downtime increase linearly with disk usage. The downtime is \u00a0over 70% of the migration time in most of the cases. We observed that on average each 1 GB of used disk space results in an increase in migration time of 18 s.<\/p>\n\n\n\n\n\n
Used DS [GB]<\/b><\/td>\n0.8<\/b><\/td>\n1.8<\/b><\/td>\n2.8<\/b><\/td>\n3.8<\/b><\/td>\n4.8<\/b><\/td>\n5.8<\/b><\/td>\n<\/tr>\n
Migration time<\/b><\/td>\n24<\/td>\n41<\/td>\n62<\/td>\n78<\/td>\n97<\/td>\n114<\/td>\n<\/tr>\n
Down time<\/b><\/td>\n15<\/td>\n31<\/td>\n46<\/td>\n62<\/td>\n73<\/td>\n85<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

[Table 1 – BLM migration & down time with increasing disk usage]<\/p>\n

\"blm-migtime-downtime\"<\/a>
Chart 1 – BLM migration & down time with increasing disk usage<\/figcaption><\/figure>\n

 <\/p>\n

Live Migration<\/b><\/p>\n

In the case of LM the migration time also increases linearly with disk usage. The down time, in contrast with BLM, stays permanently less than 1 s in every single testing scenario. In our case the LM duration increases on average by 9 s per 1 GB of used disk space.<\/p>\n\n\n\n\n\n
Used DS [GB]<\/b><\/td>\n0.8<\/b><\/td>\n1.8<\/b><\/td>\n2.8<\/b><\/td>\n3.8<\/b><\/td>\n4.8<\/b><\/td>\n5.8<\/b><\/td>\n<\/tr>\n
Migration time<\/b><\/td>\n17<\/td>\n34<\/td>\n43<\/td>\n52<\/td>\n61<\/td>\n70<\/td>\n<\/tr>\n
Down time<\/b><\/td>\n0.7<\/td>\n0.7<\/td>\n0.7<\/td>\n0.7<\/td>\n0.7<\/td>\n0.8<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

[Table 2 – LM migration & down time with increasing disk usage]<\/p>\n

\"lm-migtime-downtime\"<\/a>
Chart 2 – LM migration & down time with increasing disk usage<\/figcaption><\/figure>\n

 <\/p>\n

Comparison of LM & BLM<\/strong><\/p>\n

In all tests LM is faster than BLM. This difference increases from 41% (0.8GB used disk space) to 62% (5.8GB used disk space). According to chart 3 this performance gap will grow even more with increasing disk usage. A VM with no additional files was migrated using BLM in 24 s and in 17 s using LM. With 5GB of additional files the same machines were migrated in 144 s (BLM) and 70 s (LM).<\/p>\n\n\n\n\n\n
Used DS [GB]<\/b><\/td>\n0.8<\/b><\/td>\n1.8<\/b><\/td>\n2.8<\/b><\/td>\n3.8<\/b><\/td>\n4.8<\/b><\/td>\n5.8<\/b><\/td>\n<\/tr>\n
LM<\/b><\/td>\n17<\/td>\n34<\/td>\n43<\/td>\n52<\/td>\n61<\/td>\n70<\/td>\n<\/tr>\n
BLM<\/b><\/td>\n24<\/td>\n41<\/td>\n62<\/td>\n78<\/td>\n97<\/td>\n114<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

[Table 3 – Comparison of migration times of BLM and LM with increasing disk usage]<\/p>\n

\"migtime-blmVSLM\"<\/a>
Chart 3 – Comparison of migration times of BLM and LM with increasing disk usage<\/figcaption><\/figure>\n

 <\/p>\n

We\u2019ve already seen that LM downtime does not increase no matter what the VM \/ environment load. Now we also see, that not even the different disk usage affects this value, which stays below 1 s. In case of BLM the down time increases from 15 s (0.8 GB of used disk space) to 85 s (5.8 GB of used disk space).<\/p>\n\n\n\n\n\n
Used DS [GB]<\/b><\/td>\n0.8<\/b><\/td>\n1.8<\/b><\/td>\n2.8<\/b><\/td>\n3.8<\/b><\/td>\n4.8<\/b><\/td>\n5.8<\/b><\/td>\n<\/tr>\n
LM<\/b><\/td>\n0.7<\/td>\n0.7<\/td>\n0.7<\/td>\n0.7<\/td>\n0.7<\/td>\n0.8<\/td>\n<\/tr>\n
BLM<\/b><\/td>\n15<\/td>\n31<\/td>\n46<\/td>\n62<\/td>\n73<\/td>\n85<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

[Table 4 – Comparison of down times of BLM and LM with increasing disk usage]<\/p>\n

\"downtime-blmVSlm\"<\/a>
Chart 4 – Comparison of down times of BLM and LM with increasing disk usage<\/figcaption><\/figure>\n

 <\/p>\n

Not surprisingly the amount of transferred data grows with higher disk usage. More interesting is the fact that BLM transfers in average 1GB of data with each consumed 1GB, while BLM transfers twice as much data per one used GB – 2 GB transferred \/ 1GB consumed (it\u2019s obvious that considerable amount of these data is duplicated). See chart 5 to compare amount of data transferred via the network with different disk usage.\u00a0<\/strong><\/p>\n\n\n\n\n\n
Used DS [GB]<\/b><\/td>\n0.8<\/b><\/td>\n1.8<\/b><\/td>\n2.8<\/b><\/td>\n3.8<\/b><\/td>\n4.8<\/b><\/td>\n5.8<\/b><\/td>\n<\/tr>\n
LM<\/b><\/td>\n0.6<\/td>\n2.6<\/td>\n3.7<\/td>\n4.7<\/td>\n5.7<\/td>\n6.8<\/td>\n<\/tr>\n
BLM<\/b><\/td>\n0.8<\/td>\n2.8<\/td>\n4.9<\/td>\n6.9<\/td>\n8.9<\/td>\n10.8<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

[Table 5 – Comparison of network transfers of BLM and LM with increasing disk usage]<\/p>\n

\"data-blmVSlm\"<\/a>
Chart 5 – Comparison of network transfers of BLM and LM with increasing disk usage<\/figcaption><\/figure>\n

 <\/p>\n

Take aways<\/b><\/p>\n