Tag: neutron

Networking and Security in an Openstack Compute Node: a complex combination of iptables and (linux and OVS) bridging…

We had to investigate the operation of one of our Openstack compute nodes as it was exhibiting some unusual behaviour. We quickly determined that there was some unexpected packet loss and we had reason to believe that this could have been due to the packet processing in the node. Investigating this problem necessitated some deeper exploration of how packets are processed in the node, particularly relating to the mix of ovs bridges, linux bridges and iptables. It turns out that this is rather complex and clear information describing how all this fits together in detail is not readily available. Here, we note what we learnt from this exploration.

Continue reading

Getting Networking up and running in Openstack Cells – the Neutron way

In our previous blog post we described our experience enabling floating ips through modifications to nova python libraries in an Openstack Cells deployment using nova network. That solution was not robust enough and hence we had a go at installing neutron networking, although there is very little documentation specifically addressing Neutron and Cells. Neutron’s configuration offers better support and integration with the Cells architecture than we expected; unlike nova-network operations such as floating IP association succeed without any modifications to the source code. Here, we present an overview of the neutron networking architecture in Cells as well as main takeaways we learnt from installing it in our (small) Cells deployment. Continue reading

A Design Draft for Tenant Isolation without Tunneling in Openstack

The Problem

Cloud networking bases on tech and protocols that were not initially designed for it. This has lead to unnecessary overhead and complexity in all phases of a cloud service. Tunneling protocols generate inherent cascading and encapsulation especially in multi tenant systems. The problem increases by vendor specific configuration requirements and heterogenous architectures. This complexity leads to systems which are hard to reason about, prone to errors, energy inefficient and increases the difficulty of configuration and maintenance. Continue reading

Floating IPs management in Openstack

Openstack is generally well suited for typical use cases and there is hardly reasons to tinker with advance options and features available. Normally you would plan your public IP addresses usage and management well in advance, but if you are an experimental lab like ours, many a times things are handled in an ad-hoc manner. Recently, we ran into a unique problem which took us some time to figure out a solution.

We manage a full 160.xxx.xxx.xxx/24 block of 255 public IP addresses. Due to an underestimated user demand forecast, in our external cloud we ended up with a floating-ip pool that was woefully inadequate. One solution was to remove the external network altogether and recreate a new one with the larger floating-ip pool. The challenge was – we had real users, with experiments running on our cloud and destroying the external network was not an option.

So here is what we did to add more floating ips to the pool without even stopping or restarting any of the neutron services –

  1. Log onto your openstack controller node
  2. Read the neutron configuration file (usually located at /etc/neutron/neutron.conf
  3. Locate the connection string – this will tell you where the neutron database in located
  4. Depending on the database type (mysql, sqlite) use appropriate database managers (ours was using sqlite)

I will next show you what to do to add more IPs to the floating pool for sqlite3, this can be easily adapted for mysql.

$ sqlite3 /var/lib/neutron/ovs.sqlite
SQLite version 3.7.9 2011-11-01 00:52:41
Enter ".help" for instructions
Enter SQL statements terminated with a ";"
sqlite> .tables

The list of tables used by neutron dumped by the previous command will be similar to –

agents ovs_tunnel_endpoints
allowedaddresspairs ovs_vlan_allocations
dnsnameservers portbindingports
externalnetworks ports
extradhcpopts quotas
floatingips routerl3agentbindings
ipallocationpools routerroutes
ipallocations routers
ipavailabilityranges securitygroupportbindings
networkdhcpagentbindings securitygrouprules
networks securitygroups
ovs_network_bindings subnetroutes
ovs_tunnel_allocations subnets

The tables that are of interest to us are –

  • ipallocationpools
  • ipavailabilityranges

Next look into the schema of these tables, this will shed more light into what needs to be modified –

sqlite> .schema ipavailabilityranges
CREATE TABLE ipavailabilityranges (
allocation_pool_id VARCHAR(36) NOT NULL,
first_ip VARCHAR(64) NOT NULL,
last_ip VARCHAR(64) NOT NULL,
PRIMARY KEY (allocation_pool_id, first_ip, last_ip),
FOREIGN KEY(allocation_pool_id) REFERENCES ipallocationpools (id) ON DELETE CASCADE
sqlite> .schema ipallocationpools
CREATE TABLE ipallocationpools (
subnet_id VARCHAR(36),
first_ip VARCHAR(64) NOT NULL,
last_ip VARCHAR(64) NOT NULL,

Next look into the content of these tables, for brevity only partial outputs are shown below. Also I have masked some of the IP addresses with xxx, replace these with real values when using this guide.

sqlite> select * from ipallocationpools;
sqlite> select * from ipavailabilityranges;

Looking at the above two outputs, it is immediately clear what needs to be done next in order to add more IPs to the floating-ip range.

  1. modify the floating-ip record in the ipallocationpools table, extend the first_ip and/or last_ip value(s)
  2. for each new ip address to be added in the pool, create an entry in the ipavailabilityranges table with first_ip same as last_ip value (set to the actual IP address)

An an example, say I want to extend my pool from 160.xxx.xxx.80 to 160.xxx.xxx.82, this is what I would do

sqlite> update ipallocationpools set last_ip='160.xxx.xxx.82' where first_ip='160.xxx.xxx.32';
sqlite> insert into ipavailabilityranges values ('7bca3261-e578-4cfa-bba1-51ba6eae7791', '160.xxx.xxx.81', '160.xxx.xxx.81');
sqlite> insert into ipavailabilityranges values ('7bca3261-e578-4cfa-bba1-51ba6eae7791', '160.xxx.xxx.82', '160.xxx.xxx.82');
sqlite> .exit

And that’s all, you have 2 additional IPs available for use from your floating-ip pool. And you don’t even need to restart any of the neutron services. make sure that the subnet id is the same as in the ipallocationpools table entry.