Getting Networking up and running in Openstack Cells – the Neutron way

In our previous blog post we described our experience enabling floating ips through modifications to nova python libraries in an Openstack Cells deployment using nova network. That solution was not robust enough and hence we had a go at installing neutron networking, although there is very little documentation specifically addressing Neutron and Cells. Neutron’s configuration offers better support and integration with the Cells architecture than we expected; unlike nova-network operations such as floating IP association succeed without any modifications to the source code. Here, we present an overview of the neutron networking architecture in Cells as well as main takeaways we learnt from installing it in our (small) Cells deployment. Continue reading

Openstack Cells and nova-network: Enabling floating ip association

In our previous blog post we presented an overview of Nova Cells describing its architecture and how a basic configuration can be set up. After some further investigation it is clear why this is still considered experimental and unstable; some basic operations are not supported as yet e.g. floating ip association as well as inconsistencies in management of security groups between API and Compute Cells. Here, we focused on using only the key projects in OpenStack i.e nova, glance and keystone and avoided adding extra complexity to the system; for this reason legacy networking (nova-network) was chosen instead of Neutron – Neutron is generally more complex and we had seen problems reported with between neutron and cells. In this blog post we describe our experience enabling floating ips in an Openstack Cells architecture using nova network which required making some small modifications to the nova python libraries.

Continue reading

Floating IPs management in Openstack

Openstack is generally well suited for typical use cases and there is hardly reasons to tinker with advance options and features available. Normally you would plan your public IP addresses usage and management well in advance, but if you are an experimental lab like ours, many a times things are handled in an ad-hoc manner. Recently, we ran into a unique problem which took us some time to figure out a solution.

We manage a full 160.xxx.xxx.xxx/24 block of 255 public IP addresses. Due to an underestimated user demand forecast, in our external cloud we ended up with a floating-ip pool that was woefully inadequate. One solution was to remove the external network altogether and recreate a new one with the larger floating-ip pool. The challenge was – we had real users, with experiments running on our cloud and destroying the external network was not an option.

So here is what we did to add more floating ips to the pool without even stopping or restarting any of the neutron services –

  1. Log onto your openstack controller node
  2. Read the neutron configuration file (usually located at /etc/neutron/neutron.conf
  3. Locate the connection string – this will tell you where the neutron database in located
  4. Depending on the database type (mysql, sqlite) use appropriate database managers (ours was using sqlite)

I will next show you what to do to add more IPs to the floating pool for sqlite3, this can be easily adapted for mysql.

$ sqlite3 /var/lib/neutron/ovs.sqlite
SQLite version 3.7.9 2011-11-01 00:52:41
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables

The list of tables used by neutron dumped by the previous command will be similar to –

agents ovs_tunnel_endpoints
allowedaddresspairs ovs_vlan_allocations
dnsnameservers portbindingports
externalnetworks ports
extradhcpopts quotas
floatingips routerl3agentbindings
ipallocationpools routerroutes
ipallocations routers
ipavailabilityranges securitygroupportbindings
networkdhcpagentbindings securitygrouprules
networks securitygroups
ovs_network_bindings subnetroutes
ovs_tunnel_allocations subnets

The tables that are of interest to us are –

  • ipallocationpools
  • ipavailabilityranges

Next look into the schema of these tables, this will shed more light into what needs to be modified –

sqlite> .schema ipavailabilityranges
CREATE TABLE ipavailabilityranges (
allocation_pool_id VARCHAR(36) NOT NULL,
first_ip VARCHAR(64) NOT NULL,
last_ip VARCHAR(64) NOT NULL,
PRIMARY KEY (allocation_pool_id, first_ip, last_ip),
FOREIGN KEY(allocation_pool_id) REFERENCES ipallocationpools (id) ON DELETE CASCADE
);
sqlite> .schema ipallocationpools
CREATE TABLE ipallocationpools (
id VARCHAR(36) NOT NULL,
subnet_id VARCHAR(36),
first_ip VARCHAR(64) NOT NULL,
last_ip VARCHAR(64) NOT NULL,
PRIMARY KEY (id),
FOREIGN KEY(subnet_id) REFERENCES subnets (id) ON DELETE CASCADE
);
sqlite>

Next look into the content of these tables, for brevity only partial outputs are shown below. Also I have masked some of the IP addresses with xxx, replace these with real values when using this guide.

sqlite> select * from ipallocationpools;
b5a7b8b4-ad10-4d92-b877-e406df8ceb91|f0034b20-3566-4f9f-a6d5-b725c02f98fc|10.10.10.2|10.10.10.254
7bca3261-e578-4cfa-bba1-51ba6eae7791|765adcdf-72a4-4e07-8860-f443c7b9098b|160.xxx.xxx.32|160.xxx.xxx.80
a9994f70-2b9a-45f3-b5db-31ccc6cb7e90|72250c58-5fda-4d1b-a847-b71b432ea218|10.10.1.2|10.10.1.254
23032620-731a-4092-9509-7591b53b5ddf|12849c1f-4456-4fc1-bea6-444cce4f1ac6|10.10.2.2|10.10.2.254
fcf22336-2bd6-4e1c-92cd-e33af0b23ad9|bcf1082d-50d5-4ebc-a311-7e0618096356|10.10.11.2|10.10.11.254
bc961a47-4902-4ca2-b4f4-c5fd581a364e|09b79d08-aa92-4b99-b1fd-61d5f31d3351|10.10.25.2|10.10.25.254
sqlite> select * from ipavailabilityranges;
b5a7b8b4-ad10-4d92-b877-e406df8ceb91|10.10.10.6|10.10.10.254
a9994f70-2b9a-45f3-b5db-31ccc6cb7e90|10.10.1.2|10.10.1.2
7bca3261-e578-4cfa-bba1-51ba6eae7791|160.xxx.xxx.74|160.xxx.xxx.74
7bca3261-e578-4cfa-bba1-51ba6eae7791|160.xxx.xxx.75|160.xxx.xxx.75

Looking at the above two outputs, it is immediately clear what needs to be done next in order to add more IPs to the floating-ip range.

  1. modify the floating-ip record in the ipallocationpools table, extend the first_ip and/or last_ip value(s)
  2. for each new ip address to be added in the pool, create an entry in the ipavailabilityranges table with first_ip same as last_ip value (set to the actual IP address)

An an example, say I want to extend my pool from 160.xxx.xxx.80 to 160.xxx.xxx.82, this is what I would do

sqlite> update ipallocationpools set last_ip='160.xxx.xxx.82' where first_ip='160.xxx.xxx.32';
sqlite> insert into ipavailabilityranges values ('7bca3261-e578-4cfa-bba1-51ba6eae7791', '160.xxx.xxx.81', '160.xxx.xxx.81');
sqlite> insert into ipavailabilityranges values ('7bca3261-e578-4cfa-bba1-51ba6eae7791', '160.xxx.xxx.82', '160.xxx.xxx.82');
sqlite> .exit

And that’s all, you have 2 additional IPs available for use from your floating-ip pool. And you don’t even need to restart any of the neutron services. make sure that the subnet id is the same as in the ipallocationpools table entry.