Setting up Live Migration in Openstack Icehouse [Juno]

[Update 8.12.2014] Since OpenStack’s Juno release hasn’t introduced any changes regarding live migration, Juno users should be able to follow this tutorial as well as the Icehouse users. If you experience any issues let us know. The same setup can be used for newer versions of QEMU and Libvirt as well. Currently we are using QEMU 2.1.5 with Libvirt 1.2.11.

The Green IT theme here in ICCLab is working on monitoring and reducing datacenter energy consumption by leveraging Openstack’s live migration feature. We’ve already experimented a little with live migration in the Havana release (mostly with no luck), but since live migration is touted as one of the new stable features in the Icehouse release, we decided to investigate how it has evolved. This blogpost, largely based on official Openstack documentation, provides step-by-step walkthrough of how to setup and perform virtual machine live migration with servers running the Openstack Icehouse release and KVM/QEMU hypervisor with libvirt.

Virtual machine (VM) live migration is a process, where a VM instance, comprising of its states, memory and emulated devices, is moved from one hypervisor to another with ideally no downtime. It can come handy in many situations such as basic system maintenance, VM consolidation and more complex load management systems designed to reduce data center energy consumption. Following system configuration was used for our testing:

  • 3 nodes: 1 control node (node-1), 2 compute nodes (node-2, node-3)
  • Mirantis Openstack 5.0 (which contains a set of sensible deployment options and the Fuel deployment tool)
  • Openstack Icehouse release
  • Nova 2.18.1
  • QEMU 1.2.1
  • Libvirt 0.10.2

The default Openstack live migration process requires a shared file system (e.g. NFS, GlusterFS) across both source and destination computing hosts to copy the VM via disk. Openstack also supports “Block live migration” where a VM disk is copied via TCP and hence no shared file system is needed. Shared files system usually ensure better migration performance while the block migration approach provides better security due to file system separation.


System setup

1. Network configuration

Make sure all hosts (hypervisors) run in the same network/subnet.

1.1. DNS configuration

Check configuration and consistency of /etc/hosts file across all hosts.     node-1  node-1.domain.tld     node-2  node-2.domain.tld     node-3  node-3.domain.tld

1.2. Firewall configuration

Configure /etc/sysconfig/iptables file to allow libvirt listen on TCP port 16509 and don’t forget to add a record accepting KVM communication on TCP port within the range from 49152 to 49261.

-A INPUT -p tcp -m multiport --ports 16509 -m comment --comment "libvirt" -j ACCEPT
-A INPUT -p tcp -m multiport --ports 49152:49216 -m comment --comment "migration" -j ACCEPT

2. Libvirt configuration

Enable libvirt listen flag at /etc/sysconfig/libvirtd file.


Configure /etc/libvirt/libvirtd.conf file to make the hypervisor listen tcp communication with none athentication. Since authentication is set to NONE it’s strongly recommended to use SSH keys for authentication.

listen_tls = 0
listen_tcp = 1
auth_tcp = “none”

3. Nova configuration

Openstack doesn’t use real live migration mechanism as a default setting, because there is no guarantee that the migration is successful. An example of situation in which a migration never ends is one in which memory pages are dirtied faster than they are transfered to destination host.

To enable real live migration set up live_migration flag in /etc/nova/nova.conf file as follows:


Once these settings are configured, it should be possible to perform a live migration.

Live Migration Execution

First, list available VMs:

$ nova list
| ID                      | Name | Status | Power State | Networks      |
| 4cfe0dfb-f28f-43e9-.... | vm   | ACTIVE | Running     |      |

Next, show VM details and determine which host an instance running on:

nova show <VM-ID>
| Property                             | Value                          |
| OS-EXT-SRV-ATTR:host                 | node-2.domain.tld              |

After that, list the available compute hosts and choose the host you want to migrate the instance to:

$ nova host-list
| host_name         | service     | zone     |
| node-2.domain.tld | compute     | nova     |
| node-3.domain.tld | compute     | nova     |

Then, migrate the instance to new host.

For live migration using shared file system use:

$ nova live-migration <VM-ID> <DEST-HOST-NAME>

For block live migration use the same command with block_migrate flag enabled:

$ nova live-migration --block_migrate <VM-ID> <DEST-HOST-NAME>

Finally, show the VM details and check if it has been migrated successfully:

$ nova show <VM-ID>
| Property                             | Value                      |
| OS-EXT-SRV-ATTR:host                 | node-3.domain.tld          |

Note: If you don’t specify target compute node explicitly nova-scheduler chooses suitable one from available nodes automatically.

Congratulations, you’ve just migrated your VM.

If you are more interested in live migration performance in OpenStack you can check out our newer blog posts:



  1. Hi,

    I have tried to setup the openStack icehouse with three different nodes where one node acts as controller node, and other two nodes act as compute nodes. I have followed the instructions that is mentioned here to enable live block migration. But I am getting this following error:

    NoLiveMigrationForConfigDriveInLibVirt: Live migration of instances with config drives is not supported in libvirt unless libvirt instance path and drive data is shared across compute nodes.

    Thanks for your any kind of suggestion.

    • cima

      9. October 2014 at 10:25


      first of all please note, that there are 2 types of live migrations in Openstack – live migration (LM) and block live migration (BLM).
      LM requires shared file system between compute nodes and instance path should be the same in both cases (default is /var/lib/nova/instances).
      If you don’t use shared file system make sure that you are running the live migration with –block-migrate parameter (or check block migration option in Horizon). We also experienced some issues trying to BLOCK live migrate instance on SHARED storage. So I would suggest try using BLM only for NON-SHARED filesystem first.

      Otherwise you might be experiencing the bug described here and can apply released fix –

      • Hi,
        Thanks for replying, I am trying live migration with block migrate parameter without sharing any storage but libvirt is complaining about this “NoLiveMigrationForConfigDriveInLibVir”. I am wondering is there any configuration missing in libvirt or qemu?

        • cima

          9. October 2014 at 16:14

          Here are ours libvirt and QEMU settings:
          listen_tls = 0
          listen_tcp = 1
          auth_tcp = “none”


          Hope it helps.

          • Robayet Nasim

            9. October 2014 at 20:19

            Thanks a lot for the config file but I have already this parameters. But I have found the problem in the nova.conf, the parameter should be “force_config_drive = None” .

            I have one more query as you already did some tests with the live migration, I am wondering how you keep track of the total migration time? How the migration traffic is different from other traffics? The log file is generating lots of messages, so how can I pick the right time?

            Do you have some scripts to do the measurements like for calculating the downtime, total data transfer? If possible can you please share with me.

            Thanks again for your cordial response.

          • Hi,
            Thanks for the reply. It was the problem in nova.conf file, the parameter force_config_drive need to be false.

            I was wondering how can I keep track of the migration time as nova is producing lots of lines in log file and its hard to guess the correct time?

    • cima

      10. October 2014 at 23:36

      Glad that you’ve solved your problem.

      Concerning the testing scripts, since all these test were quite complex and not successful in every case, the best (fastest) option was to keep tracking the results manually. Here are the methods we used.

      Migration time:
      Migration iniciation generates following record in /var/log/nova-all.log at the DESTIONATION host – “Oct 10 12:49:04 node-x nova-nova.virt.libvirt.driver INFO: Instance launched has CPU info:…”
      END of the migration is recorded in the same file on the SOURCE node by record- “Oct 10 12:49:28 node-x nova-nova.compute.manager INFO: Migrating instance to node-y.domain.tld finished successfully.”

      You can simply get the migration time by subtracting these two values, but please note that “Instance launched” record is generated every time when any instance is spawned on the node and not only in the case of VM migration..
      You can GREP for Migration or Instance to get only these records.

      Downtime is calculated easily – you can just multiply number of lost packets by the interval between two following packets.

      Anyway, the idea of having these tests scripted is really tempting and can save some time in our future work. So I guess I’ll code it down. I’ll keep you updated.

  2. Hi,, Thanks for the article, this is helpfull. I try to follow your instruction, but I got a problem and this is log from /var/log/libvirt/libvirtd.log
    2014-12-16 04:36:08.505+0000: 1935: info : libvirt version: 1.2.2
    2014-12-16 04:36:08.505+0000: 1935: error : virNetSocketNewConnectTCP:484 : unable to connect to server at ‘compute2:16509’: No route to host
    2014-12-16 04:36:08.506+0000: 1935: error : doPeer2PeerMigrate:4040 : operation failed: Failed to connect to remote libvirt URI qemu+tcp://compute2/system: unable to connect to server at ‘compute2:16509’: No route to host

    Can you help me? thanks before.

    • cima

      16. December 2014 at 10:52

      Hi, it seems there is no route to the destination machine. Can you ping the destination host from the source (‘ping compute2’)? If not check /etc/hosts whether it contains record ‘ compute2’ (This file should contain ips and hostnames of every host) or try to use host’s ip address directly instead of the hostname in the configuration files (/etc/nova/nova.conf). Also check out the routing configuration on your hosts by the ‘route’ command. Hope it helps.

  3. Hi,
    I tried live migration
    I get an error like
    $nova live-migration bf441c35-d7e4-4ffa-926e-523690bd815d celestial7

    ERROR (ClientException): Live migration of instance bf441c35-d7e4-4ffa-926e-523690bd815d to host celestial7 failed (HTTP 500) (Request-ID: req-69bd07e6-3318-4f50-8663-217b98ba7564)

    Overview of instance :

    Remote error: libvirtError Requested operation is not valid: no CPU model specified [u’Traceback (most recent call last):\n’, u’ File “/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/”, line 142, in _dispatch_and_reply\n executo
    File “/opt/stack/nova/nova/conductor/”, line 606, in _live_migrate block_migration, disk_over_commit) File “/opt/stack/nova/nova/conductor/tasks/”, line 194, in execute return task.execute() File “/opt/stack/nova/nova/conductor/tasks/”, line 62, in execute self._check_requested_destination() File “/opt/stack/nova/nova/conductor/tasks/”, line 100, in _check_requested_destination self._call_livem_checks_on_host(self.destination) File “/opt/stack/nova/nova/conductor/tasks/”, line 142, in _call_livem_checks_on_host destination, self.block_migration, self.disk_over_commit) File “/opt/stack/nova/nova/compute/”, line 391, in check_can_live_migrate_destination disk_over_commit=disk_over_commit) File “/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/”, line 156, in call retry=self.retry) File “/usr/local/lib/python2.7/dist-packages/oslo_messaging/”, line 90, in _send timeout=timeout, retry=retry) File “/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/”, line 417, in send retry=retry) File “/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/”, line 408, in _send raise result
    Feb. 17, 2015, 8:20 a.m.

    Can anybody help me with this.

    • cima

      17. February 2015 at 11:22

      you can try to specify libvirt_cpu_mode and libvirt_cpu_model flags in nova.conf file ( as follows:
      libvirt_cpu_mode = custom
      libvirt_cpu_model = cpu64_rhel6

      Don’t forget to restart nova-compute service afterwards.
      You can also find more error related messages directly in the libvirtd.log on the source/destination hosts.
      Hope it helps.

      • Thanks a lot for the suggestion
        I found my cpu model in cpu_max.xml and did changes accordingly
        Still giving same error.

        libvirtd.log after error:

        2015-02-17 10:36:37.293+0000: 17166: warning : qemuOpenVhostNet:522 : Unable to open vhost-net. Opened so far 0, requested 1

        How do I solve this ?
        Thanks in advance.

        • and this

          error : netcfStateCleanup:109 : internal error: Attempt to close netcf state driver with open connections

    • cima

      19. February 2015 at 10:41

      Hello again, unfortunately I have never seen that behavior before. You could try to use directly model “cpu64_rhel6” or “kvm64” and try if it fits your configuration otherwise I would suggest to start looking at the logs from nova (nova-all.log and nova/nova-compute.log), libvirt (see the previous post) and QEMU (libvirt/qemu/instance/[].log) from both the source and destination. Could you please paste relevant parts of the those logs somewhere online or send it to me directly (cima[at] together with nova.conf file and your setup configuration?

  4. Hi,

    Please clarify…
    I do not have the directory /etc/sysconfig on ubuntu 14.04, is this something I create manually?
    If the answer is yes, should I then create the file /etc/sysconfig/libvirtd manually as well?


    • Hi Bobby, /etc/sysconfig is normally present in Redhat Linux distributions (RHEL, Centos, Fedora), but you are using Ubuntu which is a different operating system.

      libvirt is configured differently in Ubuntu.
      In Ubuntu the libvirtd configuration file is found under /etc/default and it is called “libvirt-bin” (instead of “libvirtd”). So your file should be “/etc/default/libvirt-bin” (instead of “/etc/sysconfig/libvirtd”).
      In this file you should add:
      libvirtd_opts=”-d -l”
      (Instead of:

      The rest of the process should be similar.

  5. Vasubabu Kandimalla

    25. August 2015 at 8:21

    Hi Cima,

    I am running Devstack multinode setup(one controller,2 compute) from master branch.
    I am trying to do block migration with the steps provided by you, but i am not seeing the instance migration.
    can you please help me to resolve this.
    I am observing that my nova-compute service on destination node is going offline after triggering block migration.
    Here are my setup details.
    One VM running controller+network+compute node (i had disabled nova-compute on this node).
    Two other VMs are running as compute nodes.
    I bring up VM on 1st compute node and trying to migrate to another compute node.
    Interestingly i am not seeing error when i trigger block live migration from OpenStack dashboard, but after some time my destination compute node is going offline.

    below is my nova.conf on compute node.

    vif_plugging_timeout = 300
    vif_plugging_is_fatal = True
    linuxnet_interface_driver =
    security_group_api = neutron
    network_api_class =
    firewall_driver = nova.virt.firewall.NoopFirewallDriver
    compute_driver = libvirt.LibvirtDriver
    default_ephemeral_format = ext4
    metadata_workers = 8
    ec2_workers = 8
    osapi_compute_workers = 8
    rpc_backend = rabbit
    keystone_ec2_url =
    ec2_dmz_host =
    xvpvncproxy_host =
    novncproxy_host =
    vncserver_proxyclient_address =
    vncserver_listen =
    vnc_enabled = true
    xvpvncproxy_base_url =
    novncproxy_base_url =
    logging_exception_prefix = %(color)s%(asctime)s.%(msecs)03d TRACE %(name)s ^[[01;35m%(instance)s^[[00m
    logging_debug_format_suffix = ^[[00;33mfrom (pid=%(process)d) %(funcName)s %(pathname)s:%(lineno)d^[[00m
    logging_default_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [^[[00;36m-%(color)s] ^[[01;35m%(instance)s%(color)s%(message)s^[[00m
    logging_context_format_string = %(asctime)s.%(msecs)03d %(color)s%(levelname)s %(name)s [^[[01;36m%(request_id)s ^[[00;36m%(user_name)s %(project_name)s%(color)s] ^[[01;35m%(instance)s%(color)s%(message)s^[[00m
    #force_config_drive = True
    force_config_drive = False
    send_arp_for_ha = True
    multi_host = True
    instances_path = /opt/stack/data/nova/instances
    state_path = /opt/stack/data/nova
    s3_listen =
    metadata_listen =
    ec2_listen =
    osapi_compute_listen =
    instance_name_template = instance-%08x
    my_ip =
    s3_port = 3333
    s3_host =
    default_floating_pool = public
    force_dhcp_release = True
    dhcpbridge_flagfile = /etc/nova/nova.conf
    scheduler_driver = nova.scheduler.filter_scheduler.FilterScheduler
    rootwrap_config = /etc/nova/rootwrap.conf
    api_paste_config = /etc/nova/api-paste.ini
    allow_resize_to_same_host = True
    debug = True
    verbose = True

    connection =

    connection =

    lock_path = /opt/stack/data/nova

    enabled = false
    html5proxy_base_url =

    rabbit_userid = stackrabbit
    rabbit_password = cloud
    rabbit_hosts =
    api_servers =

    os_region_name = RegionOne

    vif_driver = nova.virt.libvirt.vif.LibvirtGenericVIFDriver
    inject_partition = -2
    live_migration_uri = qemu+ssh://stack@%s/system
    #live_migration_uri = qemu+tcp://%s/system
    use_usb_tablet = False
    cpu_mode = none
    virt_type = qemu

    url =
    region_name = RegionOne
    admin_tenant_name = service
    auth_strategy = keystone
    admin_auth_url =
    admin_password = cloud
    admin_username = neutron

    fixed_key = f1a0b6c1ce848e4ed2dad31abb3e7fd49183dc558e34931c22eb4884a9096ddc
    I am able to ssh without password through stack user from controller node to all compute nodes.

    • Vasubabu Kandimalla

      25. August 2015 at 8:24

      Correction in above comment:
      nova-compute on source compute node is going offline not destination.

  6. Vasubabu Kandimalla

    25. August 2015 at 8:35

    Hi Cima,

    Below is the output from libvirt.log
    2015-08-25 06:34:32.323+0000: 4216: debug : virConnectGetLibVersion:1590 : conn=0x7f7b14002460, libVir=0x7f7b3bde1b90
    2015-08-25 06:34:32.325+0000: 4212: debug : virDomainLookupByName:2121 : conn=0x7f7b14002460, name=instance-00000014
    2015-08-25 06:34:32.325+0000: 4212: debug : qemuDomainLookupByName:1402 : Domain not found: no domain with matching name ‘instance-00000014’
    2015-08-25 06:34:32.327+0000: 4213: debug : virConnectGetLibVersion:1590 : conn=0x7f7b14002460, libVir=0x7f7b3d5e4b90
    2015-08-25 06:34:32.328+0000: 4214: debug : virDomainLookupByName:2121 : conn=0x7f7b14002460, name=instance-00000014
    2015-08-25 06:34:32.328+0000: 4214: debug : qemuDomainLookupByName:1402 : Domain not found: no domain with matching name ‘instance-00000014’
    2015-08-25 06:34:32.612+0000: 4218: debug : virConnectGetLibVersion:1590 : conn=0x7f7b14002460, libVir=0x7f7b3addfb90
    2015-08-25 06:34:32.614+0000: 4216: debug : virDomainLookupByName:2121 : conn=0x7f7b14002460, name=instance-00000014
    2015-08-25 06:34:32.614+0000: 4216: debug : qemuDomainLookupByName:1402 : Domain not found: no domain with matching name ‘instance-00000014’

    • Vasubabu Kandimalla

      25. August 2015 at 9:00

      The above one is libvirtd.log snapshot from destination host, but i didn’t see any issue on source compute node where instance is running

      • cima

        26. August 2015 at 9:42

        Hello, I would suggest to take a look “one level up” and check nova-compute.log on both the source and the destination. Also Qemu logs could be useful (note that Qemu records logs per instance. Default location is /var/log/libvirt/qemu/[instance_name].log). Log is created after the instance is spawned. You can get the instance name using “virsh list” command on the host it runs. Virsh command also gives you an insight on the instance state and whether the instance is being created on the destination on removed from the source.

        • Vasubabu Kandimalla

          26. August 2015 at 14:25

          Hi Cima,

          There was ssh key issue, I replace ssh with tcp in nova configuration and now it is working fine.


  7. Hello,

    I am trying block migration in Juno and running into the below error:

    Starting monitoring of live migration _live_migration /usr/lib/python2.7/site-packages/nova/virt/libvirt/
    2015-09-09 14:32:03.939 12766 DEBUG nova.virt.libvirt.driver [-] [instance: 9bb19e79-a8a2-481f-aefb-e05d0567c198] Operation thread is still running _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/
    2015-09-09 14:32:03.940 12766 DEBUG nova.virt.libvirt.driver [-] [instance: 9bb19e79-a8a2-481f-aefb-e05d0567c198] Migration not running yet _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/
    2015-09-09 14:32:03.948 12766 ERROR nova.virt.libvirt.driver [-] [instance: 9bb19e79-a8a2-481f-aefb-e05d0567c198] Live Migration failure: Unable to pre-create chardev file ‘/var/lib/nova/instances/9bb19e79-a8a2-481f-aefb-e05d0567c198/console.log’: No such file or directory
    2015-09-09 14:32:03.948 12766 DEBUG nova.virt.libvirt.driver [-] [instance: 9bb19e79-a8a2-481f-aefb-e05d0567c198] Migration operation thread notification thread_finished /usr/lib/python2.7/site-packages/nova/virt/libvirt/
    2015-09-09 14:32:04.441 12766 DEBUG nova.virt.libvirt.driver [-] [instance: 9bb19e79-a8a2-481f-aefb-e05d0567c198] VM running on src, migration failed _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/
    2015-09-09 14:32:04.442 12766 DEBUG nova.virt.libvirt.driver [-] [instance: 9bb19e79-a8a2-481f-aefb-e05d0567c198] Fixed incorrect job type to be 4 _live_migration_monitor /usr/lib/python2.7/site-packages/nova/virt/libvirt/
    2015-09-09 14:32:04.442 12766 ERROR nova.virt.libvirt.driver [-] [instance: 9bb19e79-a8a2-481f-aefb-e05d0567c198] Migration operation has aborted

    It used to work well for me with the Icehouse version.Not sure whats wrong here.Am i hitting the bug mentioned below,any idea?

Leave a Reply

Your email address will not be published. Required fields are marked *