{"id":9911,"date":"2016-04-21T11:24:55","date_gmt":"2016-04-21T09:24:55","guid":{"rendered":"https:\/\/blog.zhaw.ch\/icclab\/?p=9911"},"modified":"2016-04-21T13:29:02","modified_gmt":"2016-04-21T11:29:02","slug":"experience-with-neutron-ha-in-openstack","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/","title":{"rendered":"Experience with Neutron High Availability (HA) in Openstack"},"content":{"rendered":"<p>For the Zurich <a href=\"https:\/\/www.fiware.org\/\">FIWARE<\/a> node, we&#8217;re setting up a <a href=\"https:\/\/www.openstack.org\/software\/kilo\/\">Kilo<\/a> High Availability (HA) deployment &#8211; we&#8217;re transitioning from our current <a href=\"https:\/\/www.openstack.org\/software\/icehouse\/\">Icehouse<\/a> (non-HA) deployment.<\/p>\n<p>Kilo HA is recommended as there is a general understanding within the project that the HA capabilities are now ready for production use. However, there is no single Kilo HA &#8211; there are many different configurations which can be called HA &#8211; and in this post, we describe some of the points we encountered while setting up our HA node.<\/p>\n<p>We deployed\u00a0<a href=\"https:\/\/software.mirantis.com\/releases\/\">Mirantis Openstack v7.0<\/a> using the <a href=\"https:\/\/wiki.openstack.org\/wiki\/Fuel\">Fuel<\/a> deployment tool, as is used in the project and as we have used before; requiring a HA deployment, we selected the HA configuration in Fuel and we have 3 controller nodes to provide HA. We did have some issues that the deployment did not terminate cleanly, failing in some <code><a href=\"https:\/\/github.com\/openstack\/fuel-astute\">astute<\/a><\/code>-based post-deployment tests &#8211;\u00a0however, these issues were minor and the system behaved in a sane manner.<\/p>\n<p><!--more-->The deployment seemed to work fine initially, but we made some changes and after some time it entered an unexpected state and networking functions were unstable: at this point, we reviewed the system configuration and were a little surprised at what we saw.<\/p>\n<p>But before explaining the configuration, it is first necessary to understand that <a href=\"https:\/\/wiki.openstack.org\/wiki\/Neutron\">Neutron<\/a> (Kilo) provides three types of routers:<\/p>\n<ul>\n<li>Legacy routers: these routers operate on a single neutron node and are exactly those that are used in non-HA deployments. Resilience is provided in a HA deployment by determining if the l3 agent is down using a heartbeat mechanism and if so, restarting the routers which are no longer available on alternative network nodes.<\/li>\n<li><a href=\"http:\/\/docs.openstack.org\/liberty\/networking-guide\/scenario_l3ha_lb.html\">HA routers<\/a>: these routers operate across multiple neutron nodes (having a <a href=\"http:\/\/docs.openstack.org\/developer\/neutron\/devref\/layer3.html\">qrouter<\/a> on each node) and operate in an active\/passive mode with one of the neutron nodes hosting the active router &#8211; <a href=\"https:\/\/en.wikipedia.org\/wiki\/Virtual_Router_Redundancy_Protocol\">VRRP<\/a> and <a href=\"http:\/\/www.keepalived.org\/\">keepalived<\/a> is used to pass control to another router in case of failure<\/li>\n<li><a href=\"https:\/\/wiki.openstack.org\/wiki\/Neutron\/DVR\">Distributed Virtual Routers<\/a> (DVRs): these are more complex routers which also operate across multiple neutron nodes however, in this case, the load is distributed evenly across all routers during operation and not just a single active router<\/li>\n<\/ul>\n<p>The view within the project is that DVR is still a little too experimental for production scenarios and consequently the options are legacy or HA routers.<\/p>\n<p>To our surprise, the HA Fuel deployment employed legacy routers and not HA routers &#8211; in <code>neutron.conf<\/code>, the following configurations were set:<\/p>\n<pre>l3_ha = false\r\nallow_automatic_l3agent_failover = true\r\n<\/pre>\n<p>We experimented with the use of HA routers instead of legacy routers as these should result in a more rapid failover since the router config is already present on all nodes. Changing the <code>l3_ha<\/code> parameter to true in <code>neutron.conf<\/code> facilitates this. (Note that it is also possible to have neutron create legacy routers by default but to specify at router creation time that a router should be HA router using <code>neutron router-create --ha=True<\/code> so the <code>neutron.conf<\/code> parameter is not absolute).<\/p>\n<p>We did have significant problems with the HA routers, however. We found that they were not generating <a href=\"http:\/\/networkengineering.stackexchange.com\/questions\/7713\/how-does-gratuitous-arp-work\">gratuitous ARPs<\/a> and sending them to our gateway when a new floating IP address was assigned. We used a mix of the standard <a href=\"http:\/\/www.tcpdump.org\/\">tcpdump<\/a> and <a href=\"http:\/\/blog.scottlowe.org\/2013\/09\/04\/introducing-linux-network-namespaces\/\">ip netns<\/a>\u00a0tools on both linux bridges and <a href=\"http:\/\/openvswitch.org\/\">OVS<\/a> bridge to\u00a0poke around inside neutron to see what was going on. The essential test was to check what was going out on <code>br-ex<\/code> &#8211; we saw no G-ARPs going out on this interface when a floating IP was assigned, but we also observed issues inside other bridges (<code>br-int<\/code> specifically) in which there were mismatches between assigned IP addresses and MAC addresses). In a test\/pre-deploy environment in which floating IPs were changing quite dynamically, this resulted in our gateway quickly losing track of the mapping from floating IP to appropriate MAC address resulting in lost connectivity.<\/p>\n<p>Ultimately, we decided to stop working with HA routers and reverted to legacy routers. For our modest\u00a0context, the legacy neutron router\u00a0failed over in about a minute when we shut down one of our neutron nodes &#8211; this is good enough for us for now. In future, we will explore the alternative options as they mature.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>For the Zurich FIWARE node, we&#8217;re setting up a Kilo High Availability (HA) deployment &#8211; we&#8217;re transitioning from our current Icehouse (non-HA) deployment. Kilo HA is recommended as there is a general understanding within the project that the HA capabilities are now ready for production use. However, there is no single Kilo HA &#8211; there [&hellip;]<\/p>\n","protected":false},"author":92,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[1],"tags":[],"features":[],"class_list":["post-9911","post","type-post","status-publish","format-standard","hentry","category-allgemein"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Experience with Neutron High Availability (HA) in Openstack - Service Engineering (ICCLab &amp; SPLab)<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Experience with Neutron High Availability (HA) in Openstack\" \/>\n<meta property=\"og:description\" content=\"For the Zurich FIWARE node, we&#8217;re setting up a Kilo High Availability (HA) deployment &#8211; we&#8217;re transitioning from our current Icehouse (non-HA) deployment. Kilo HA is recommended as there is a general understanding within the project that the HA capabilities are now ready for production use. However, there is no single Kilo HA &#8211; there [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/\" \/>\n<meta property=\"og:site_name\" content=\"Service Engineering (ICCLab &amp; SPLab)\" \/>\n<meta property=\"article:published_time\" content=\"2016-04-21T09:24:55+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-04-21T11:29:02+00:00\" \/>\n<meta name=\"author\" content=\"Sean Murphy\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sean Murphy\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/\"},\"author\":{\"name\":\"Sean Murphy\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/c87a6eef7e1f4a152aeec5f8b9527b8d\"},\"headline\":\"Experience with Neutron High Availability (HA) in Openstack\",\"datePublished\":\"2016-04-21T09:24:55+00:00\",\"dateModified\":\"2016-04-21T11:29:02+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/\"},\"wordCount\":691,\"commentCount\":0,\"articleSection\":[\"*.*\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/\",\"name\":\"Experience with Neutron High Availability (HA) in Openstack - Service Engineering (ICCLab &amp; SPLab)\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\"},\"datePublished\":\"2016-04-21T09:24:55+00:00\",\"dateModified\":\"2016-04-21T11:29:02+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/c87a6eef7e1f4a152aeec5f8b9527b8d\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/icclab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Experience with Neutron High Availability (HA) in Openstack\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/\",\"name\":\"Service Engineering (ICCLab &amp; SPLab)\",\"description\":\"A Blog of the ZHAW Zurich University of Applied Sciences\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/c87a6eef7e1f4a152aeec5f8b9527b8d\",\"name\":\"Sean Murphy\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/4514cb0ddfe236fd05d5ddb715bc19e1e1e35dafa16bb1b911e6094d278211d6?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/4514cb0ddfe236fd05d5ddb715bc19e1e1e35dafa16bb1b911e6094d278211d6?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/4514cb0ddfe236fd05d5ddb715bc19e1e1e35dafa16bb1b911e6094d278211d6?s=96&d=mm&r=g\",\"caption\":\"Sean Murphy\"},\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/author\/murp\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Experience with Neutron High Availability (HA) in Openstack - Service Engineering (ICCLab &amp; SPLab)","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/","og_locale":"en_US","og_type":"article","og_title":"Experience with Neutron High Availability (HA) in Openstack","og_description":"For the Zurich FIWARE node, we&#8217;re setting up a Kilo High Availability (HA) deployment &#8211; we&#8217;re transitioning from our current Icehouse (non-HA) deployment. Kilo HA is recommended as there is a general understanding within the project that the HA capabilities are now ready for production use. However, there is no single Kilo HA &#8211; there [&hellip;]","og_url":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/","og_site_name":"Service Engineering (ICCLab &amp; SPLab)","article_published_time":"2016-04-21T09:24:55+00:00","article_modified_time":"2016-04-21T11:29:02+00:00","author":"Sean Murphy","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Sean Murphy","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/"},"author":{"name":"Sean Murphy","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/c87a6eef7e1f4a152aeec5f8b9527b8d"},"headline":"Experience with Neutron High Availability (HA) in Openstack","datePublished":"2016-04-21T09:24:55+00:00","dateModified":"2016-04-21T11:29:02+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/"},"wordCount":691,"commentCount":0,"articleSection":["*.*"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/","url":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/","name":"Experience with Neutron High Availability (HA) in Openstack - Service Engineering (ICCLab &amp; SPLab)","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#website"},"datePublished":"2016-04-21T09:24:55+00:00","dateModified":"2016-04-21T11:29:02+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/c87a6eef7e1f4a152aeec5f8b9527b8d"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/icclab\/experience-with-neutron-ha-in-openstack\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/icclab\/"},{"@type":"ListItem","position":2,"name":"Experience with Neutron High Availability (HA) in Openstack"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/icclab\/#website","url":"https:\/\/blog.zhaw.ch\/icclab\/","name":"Service Engineering (ICCLab &amp; SPLab)","description":"A Blog of the ZHAW Zurich University of Applied Sciences","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/c87a6eef7e1f4a152aeec5f8b9527b8d","name":"Sean Murphy","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/4514cb0ddfe236fd05d5ddb715bc19e1e1e35dafa16bb1b911e6094d278211d6?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/4514cb0ddfe236fd05d5ddb715bc19e1e1e35dafa16bb1b911e6094d278211d6?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/4514cb0ddfe236fd05d5ddb715bc19e1e1e35dafa16bb1b911e6094d278211d6?s=96&d=mm&r=g","caption":"Sean Murphy"},"url":"https:\/\/blog.zhaw.ch\/icclab\/author\/murp\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/9911","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/users\/92"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/comments?post=9911"}],"version-history":[{"count":10,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/9911\/revisions"}],"predecessor-version":[{"id":9945,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/9911\/revisions\/9945"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/media?parent=9911"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/categories?post=9911"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/tags?post=9911"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/features?post=9911"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}