{"id":6794,"date":"2014-11-20T15:37:37","date_gmt":"2014-11-20T13:37:37","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=6794"},"modified":"2016-05-23T10:48:32","modified_gmt":"2016-05-23T08:48:32","slug":"post-copy-live-migration-in-qemu","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/","title":{"rendered":"Post-copy live migration in QEMU"},"content":{"rendered":"

Hurray! We have finally deployed QEMU<\/a> 2.1.5 with post-copy<\/a> live migration support on our servers! But before we get to that, a little bit of context… in our previous blog <\/a>posts we focused on performance analysis of pre-copy live migration in Openstack<\/a>. So far all of our experiments were done using QEMU<\/a> version 1.2 with KVM<\/a> acceleration. As we were keen to do some experimentation with post-copy live migration, we had to upgrade to the very new QEMU 2.1.5 which provides post-copy live migration support in one of its branches. (More generally, there have been significant enhancements in QEMU since version 1.2 \u00a0– of November 2012 – and hence we expected better performance and reliability in pre-copy as well). This blog post focuses on our first hands-on experience with post-copy live migration in QEMU.<\/p>\n

<\/p>\n

The main difference between post-copy<\/a> and pre-copy<\/a> migration is the way in which the VM\u2019s RAM is transferred. While pre-copy spawns a new instance and copies all the memory pages before the old one is turned off, post-copy spawns and runs the new instance immediately and fetches all the missing memory pages as needed.<\/p>\n

In our earlier experiments, we observed that the pre-copy approach fails to live migrate instances that contain intensive memory activity. More precisely, the rate of change of memory in the VM MUST be lower than the throughput of the network interface which is used for the migration to achieve successful convergence. Post-copy live migration enables live migration to terminate in finite time irrespective of VM memory load thus providing a \u00a0reliable tool to manage server load between compute hosts in any circumstances.<\/p>\n

Note that in order to use post-copy the kernel needs to support userfaultfd<\/i><\/a> and remap_anon_pages<\/i><\/a> syscalls which are necessary to handle memory page faults in user space. In the post-copy live migration context it\u2019s used as follows – the newly spawned VM tries to access a memory page and if it fails, the VM fetches the memory page from original \u00a0VM memory over the network. These syscalls however haven\u2019t been merged into the official kernel release yet, but you can use the available patch<\/a> by Andrea Arcangeli. After quite some effort we managed to get Ubuntu<\/a> 12.10 LTS running on customized 3.18.0-rc3+ kernel with this patch applied.<\/p>\n

Using post-copy live migration in QEMU<\/b><\/p>\n

So, how can it be used? You can use either the QEMU monitor directly by pressing alt+2 in instance\u2019s VNC session (alt+1 to switch back) or if you are using libvirt you can use virsh qemu-monitor-command [domain] –hmp \u2018[command]\u2019<\/i> command instead of using qemu monitor directly. Just substitute the [domain]<\/i> with the one you want to access listed under virsh list<\/i>.<\/p>\n

1. Enable post-copy capability for the domain (instance) you want to migrate BEFORE the migration initialization.<\/p>\n

QEMU < 2.5 (experimental):<\/p>\n

(qemu) migrate_set_capability x-postcopy-ram on<\/code><\/p>\n

QEMU >= 2.5:<\/p>\n

(qemu) migrate_set_capability postcopy-ram on<\/code><\/p>\n

Note, you can check the status of all migration capabilities by:<\/p>\n

(qemu) info migrate_capabilities<\/code><\/p>\n

2. Initialize live migration as usual (standard pre-copy mechanism is used)<\/p>\n

(qemu) virsh migrate --live [domain] qemu+ssh:\/\/[host-ip]\/system<\/code><\/p>\n

3. Once the live migration is initialized you can switch to post-copy (not before)<\/p>\n

(qemu) migrate_start_postcopy<\/code><\/p>\n

Note that, somewhat confusingly, the post-copy mechanism is not used by default even after you enable the post-copy capability. This flag ensures that that the destination system provides all the needed calls and functions to support post-copy mode and if not, the live migration is not initialized. The switch to post-copy migration needs to be done after migration initialization. As such, the approach is neither pure pre-copy nor pure post-copy, but a hybrid which combines aspects of both. The main reason for this hybrid live migration approach is that in many use cases the VM can be successfully migrated with the pre-copy technique without the higher risk of losing an instance if a network failure occurs. Switching to the post-copy can be done at any time if it is concluded that the \u00a0migration is taking too long and it probably won\u2019t converge in reasonable time (or at all) and therefore it can be used as a fallback for scenarios where the VM is stuck in a migrating state due to its high memory activity. The main drawback of the post-copy mechanism is that any network interruption causes loosing of the VM.<\/p>\n

Since this is a very new feature, there is a lot of work being done to support post-copy live migration in libvirt<\/a>\u2019s API and even more work to be done to support this functionality in Openstack<\/a>, particularly within Nova.<\/p>\n

The combination of these more powerful live migration solutions offers the potential for much more fluid and flexible load management in data centers although the capabilities and performance is still not well understood. Therefore our next work will focus on getting the post-copy live migration to work in an OpenStack context.<\/p>\n

[UPDATE 23.5.2016]<\/strong><\/p>\n

As pointed out by Md Haris Iqbal\u00a0in the comment section (thanks!), the postcopy capability\u00a0is no longer experimental in QEMU version 2.5 and higher. Therefore, to enable it in such cases simply use:<\/p>\n

(qemu) migrate_set_capability postcopy-ram on<\/code><\/p>\n

(without the “x-” prefix)<\/p>\n

Schlagw\u00f6rter: libvirt<\/a>, live migration<\/a>, post-copy<\/a>, qemu<\/a>, remap_anon_pages<\/a>, userfaultfd<\/a>
<\/div>","protected":false},"excerpt":{"rendered":"

Hurray! We have finally deployed QEMU 2.1.5 with post-copy live migration support on our servers! But before we get to that, a little bit of context… in our previous blog posts we focused on performance analysis of pre-copy live migration in Openstack. So far all of our experiments were done using QEMU version 1.2 with […]<\/p>\n","protected":false},"author":155,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[5],"tags":[515,450,516,514,518,517],"features":[],"class_list":["post-6794","post","type-post","status-publish","format-standard","hentry","category-articles","tag-libvirt","tag-live-migration","tag-post-copy","tag-qemu","tag-remap_anon_pages","tag-userfaultfd"],"yoast_head":"\nPost-copy live migration in QEMU - Service Engineering (ICCLab & SPLab)<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Post-copy live migration in QEMU\" \/>\n<meta property=\"og:description\" content=\"Hurray! We have finally deployed QEMU 2.1.5 with post-copy live migration support on our servers! But before we get to that, a little bit of context… in our previous blog posts we focused on performance analysis of pre-copy live migration in Openstack. So far all of our experiments were done using QEMU version 1.2 with […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/\" \/>\n<meta property=\"og:site_name\" content=\"Service Engineering (ICCLab & SPLab)\" \/>\n<meta property=\"article:published_time\" content=\"2014-11-20T13:37:37+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-05-23T08:48:32+00:00\" \/>\n<meta name=\"author\" content=\"cima\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"cima\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/\"},\"author\":{\"name\":\"cima\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/bfb29453d69d6e8260cbfc3ff86df20b\"},\"headline\":\"Post-copy live migration in QEMU\",\"datePublished\":\"2014-11-20T13:37:37+00:00\",\"dateModified\":\"2016-05-23T08:48:32+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/\"},\"wordCount\":824,\"commentCount\":4,\"keywords\":[\"libvirt\",\"live migration\",\"post-copy\",\"qemu\",\"remap_anon_pages\",\"userfaultfd\"],\"articleSection\":[\"Articles\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/\",\"name\":\"Post-copy live migration in QEMU - Service Engineering (ICCLab & SPLab)\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\"},\"datePublished\":\"2014-11-20T13:37:37+00:00\",\"dateModified\":\"2016-05-23T08:48:32+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/bfb29453d69d6e8260cbfc3ff86df20b\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/icclab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Post-copy live migration in QEMU\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/\",\"name\":\"Service Engineering (ICCLab & SPLab)\",\"description\":\"A Blog of the ZHAW Zurich University of Applied Sciences\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/bfb29453d69d6e8260cbfc3ff86df20b\",\"name\":\"cima\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/8d7cb1dd9511a1465510abf53f139c259c510dbce26f647ee43e3771f65a7143?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/8d7cb1dd9511a1465510abf53f139c259c510dbce26f647ee43e3771f65a7143?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/8d7cb1dd9511a1465510abf53f139c259c510dbce26f647ee43e3771f65a7143?s=96&d=mm&r=g\",\"caption\":\"cima\"},\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/author\/cima\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Post-copy live migration in QEMU - Service Engineering (ICCLab & SPLab)","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/","og_locale":"en_US","og_type":"article","og_title":"Post-copy live migration in QEMU","og_description":"Hurray! We have finally deployed QEMU 2.1.5 with post-copy live migration support on our servers! But before we get to that, a little bit of context… in our previous blog posts we focused on performance analysis of pre-copy live migration in Openstack. So far all of our experiments were done using QEMU version 1.2 with […]","og_url":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/","og_site_name":"Service Engineering (ICCLab & SPLab)","article_published_time":"2014-11-20T13:37:37+00:00","article_modified_time":"2016-05-23T08:48:32+00:00","author":"cima","twitter_card":"summary_large_image","twitter_misc":{"Written by":"cima","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/"},"author":{"name":"cima","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/bfb29453d69d6e8260cbfc3ff86df20b"},"headline":"Post-copy live migration in QEMU","datePublished":"2014-11-20T13:37:37+00:00","dateModified":"2016-05-23T08:48:32+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/"},"wordCount":824,"commentCount":4,"keywords":["libvirt","live migration","post-copy","qemu","remap_anon_pages","userfaultfd"],"articleSection":["Articles"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/","url":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/","name":"Post-copy live migration in QEMU - Service Engineering (ICCLab & SPLab)","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#website"},"datePublished":"2014-11-20T13:37:37+00:00","dateModified":"2016-05-23T08:48:32+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/bfb29453d69d6e8260cbfc3ff86df20b"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/icclab\/post-copy-live-migration-in-qemu\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/icclab\/"},{"@type":"ListItem","position":2,"name":"Post-copy live migration in QEMU"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/icclab\/#website","url":"https:\/\/blog.zhaw.ch\/icclab\/","name":"Service Engineering (ICCLab & SPLab)","description":"A Blog of the ZHAW Zurich University of Applied Sciences","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/bfb29453d69d6e8260cbfc3ff86df20b","name":"cima","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/8d7cb1dd9511a1465510abf53f139c259c510dbce26f647ee43e3771f65a7143?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/8d7cb1dd9511a1465510abf53f139c259c510dbce26f647ee43e3771f65a7143?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/8d7cb1dd9511a1465510abf53f139c259c510dbce26f647ee43e3771f65a7143?s=96&d=mm&r=g","caption":"cima"},"url":"https:\/\/blog.zhaw.ch\/icclab\/author\/cima\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/6794","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/users\/155"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/comments?post=6794"}],"version-history":[{"count":12,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/6794\/revisions"}],"predecessor-version":[{"id":10046,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/6794\/revisions\/10046"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/media?parent=6794"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/categories?post=6794"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/tags?post=6794"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/features?post=6794"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}