Collecting energy consumption data using Kwapi in Openstack

In some of our projects we need to understand the energy consumption of our servers in an Openstack cluster. The first step in this process was to collect energy consumption data from our IBM servers; we stored this in Ceilometer for further study. In this blog post we will cover how we do this.

First, Kwapi 101.

Kwapi is a part of the openstack ecosystem (perhaps a little peripheral) which is focused on collecting energy data. It has pretty good integration with Ceilometer which enables the energy data to be stored there.

Kwapi is architected in such a way that individual drivers listen to a specific wattmeter – a wattmeter can be a physical energy meter with a wifi interface or connected to ipmi, i.e. any kind of device that measures energy consumption. The drivers then pass the information on to plug-ins. These are the API, the forwarder and the RRD plugin which is the visualization plugin that provides a web interface with power consumption graphs.

The API listens to the forwarder endpoint and collects data – it also provides an outbound interface to enable other entities e.g. Ceilometer to get this data. The purpose of the forwarder is to reduce network traffic by duplicating the metrics and sending only once if multiple plugins listen to the same driver; use of the forwarder is optional, If the drivers and the plugins are running on the same machine it is not necessary to use the forwarder. The architecture is shown in the figure below.

Following that  basic introduction, we now describe how we set it up.

Our IBM x-series servers (x3550 M4 to be precise) provide the energy consumed via a sysfs interface to the ibmaem module; the energy consumed is stored in a dynamically updated file that gives the number of Joules consumed since the system was powered up – the amount of energy consumed by a task can be calculated by taking the difference between before and after readings.

The commands below show how we set up the servers to collect the energy consumed. The first step was to load the ibmaem module and install lm-sensors to read the power and energy sensors.

--Install lm_sensors
# apt-get install lm-sensors
--Load the module
# modprobe ibmaem
-- Scan your system and detect hardware monitoring chips
# sensors-detect

Once these have been installed successfully the state of the system’s sensors can be seen:

[root@node-1 ~]# sensors
aem2-isa-0000
Adapter: ISA adapter
temp1:       +22.0°C
temp2:       +21.0°C
power1:     161.56 W  (interval =   1.00 s)
power2:     144.67 W  (interval =   1.00 s)
power3:      26.00 W
power4:     427.00 W
power5:     419.00 W
power6:     413.80 W
power7:     293.80 W
energy1:      1.81 GJ
energy2:      2.30 GJ

The ibmaem module provides the power and energy sensor readings where energy2 gives a readout of the rack-mounted servers and energy1 for blade servers; the file where this data is stored can be found  in /sys/devices/platform/aem.1/energyX_input, where X is 1 for blade servers and 2 for rack-mounted.

On Ubuntu setting up kwapi requires going through the following steps.

First, install some basic packages:

# apt-get install python-dev python-zmq python-rrdtool libxml2-dev libglib2.0 librrd-dev
# pip install greenlet babel prettytable stevedore netaddr

Then download and install Kwapi from the repository:

# git clone https://github.com/stackforge/kwapi.git
# cd kwapi
# ./setup.py install

NOTE: While we show the process for ubuntu above, in our main deployment we use CentOS6.5 and had issues with the package manager yum and the rrdtool dependencies – we solved it by removing rrdtool from the dependencies needed by Kwapi in the file kwapi/requirements.txt.

After installing Kwapi, copy the configurations file to /etc and create the log files, as follows:

# cp -r etc/kwapi /etc/
# mkdir /var/log/kwapi
# touch /var/log/kwapi/kwapi-api.log
# touch /var/log/kwapi/kwapi-drivers.log
# touch /var/log/kwapi/kwapi-forwarder.log

Kwapi already has some drivers pre-configured (IPMI, SNMP, Wattsup) but it also supports the addition of new drivers.; the dummy driver is a good start to have a better comprehension of how it works. To activate a driver add it to /etc/kwapi/drivers.conf with all necessary parameters.

After configuring Kwapi, start the services:

# kwapi-api
# kwapi-drivers
# kwapi-forwarder

Ceilometer already supports Kwapi, it is just necessary to tell Ceilometer where kwapi is. This involves setting up the service and endpoint in keystone as follows:

# keystone service-create --name=Kwapi \
                          --type=energy \
                          --description="Kwapi Energy Monitor"

+-------------+----------------------------------+
|   Property  |              Value               |
+-------------+----------------------------------+
| description | Kwapi Energy Monitor             |
| id          | 15c11a12312e427e91bc41231b45f4bd |
| name        | Kwapi                            |
| type        | energy                           |
+-------------+----------------------------------+

# keystone endpoint-create --region=RegionOne \
                           --service-id=service_id \
                           --publicurl=http://10.0.0.1:api_port/v1 \
                           --internalurl=http://10.0.0.1:api_port/v1 \
                           --adminurl=http://10.0.0.1:api_port/v1

+-------------+-----------------------------------+
|   Property  |             Value                 |
+-------------+-----------------------------------+
| adminurl    | http://10.0.0.1:api_port/v1       |
| id          | 11f9c625a4324a3f8e66bf4e5dadf213  |
| internalurl | http://10.0.0.1:api_port/v1       |
| publicurl   | http://10.0.0.1:api_port/v1       |
| region      | RegionOne                         |
| service_id  | service_id                        |
+-------------+-----------------------------------+

When a new service is created a service id is assigned by keystone, this id must be used in the endpoint-create call in service_id field.

Then change file /etc/ceilometer/pipeline.yaml and create a new source to recognize the energy meter – here is how we did it:

- name: meter_energy
interval: interval 
meters:
- "power*"
- "energy*"
sinks:
- meter_sink

Note that interval is the time between which ceilometer will query the API and collect the samples in seconds – an interval of 60 will collect a sample from kwapi every minute. Note that the drivers themselves have their own sampling interval, so care should be taken when configuring this to ensure that there are not significant inconsistencies between these values.

Restart the Ceilometer central agent service.

# service openstack-ceilometer-agent restart

Following these steps the energy consumption is retrieved by the Kwapi API and stored in Ceilometer – this can be seen below.

# ceilometer meter-list

+-------+-----------+-----+--------------------+-----------+-----------+
| Name  | Type      | Unit| Resource ID        | User ID   | Project ID|
+-------+-----------+-----+--------------------+-----------+-----------+
| energy| cumulative| kWh | node-41.domain.tld | None      | None      |
| power | gauge     | W   | node-41.domain.tld | None      | None      |
+-------+-----------+-----+--------------------+-----------+-----------+

We can see some of the samples below:

# sample-list -m power -q “resource=node-41.domain.tld”

+-------------------+------+--------+------+----------------------------+
| Resource ID       | Name | Volume | Unit | Timestamp                  |
+-------------------+------+--------+------+----------------------------+
| node-41.domain.tld| power| 173.0  | W    | 2014-08-14T07:25:45.023000 |
| node-41.domain.tld| power| 169.0  | W    | 2014-08-14T07:25:34.908000 |
| node-41.domain.tld| power| 169.0  | W    | 2014-08-14T07:25:24.760000 |
| node-41.domain.tld| power| 174.0  | W    | 2014-08-14T07:25:14.646000 |
| node-41.domain.tld| power| 174.0  | W    | 2014-08-14T07:25:14.646000 |
| node-41.domain.tld| power| 172.0  | W    | 2014-08-14T07:25:04.532000 |
| node-41.domain.tld| power| 168.0  | W    | 2014-08-14T07:24:54.418000 |
| node-41.domain.tld| power| 170.0  | W    | 2014-08-14T07:24:44.269000 |
| node-41.domain.tld| power| 174.0  | W    | 2014-08-14T07:24:34.154000 |
| node-41.domain.tld| power| 173.0  | W    | 2014-08-14T07:24:24.037000 |
| node-41.domain.tld| power| 173.0  | W    | 2014-08-14T07:24:24.037000 |
| node-41.domain.tld| power| 169.0  | W    | 2014-08-14T07:24:13.922000 |
| node-41.domain.tld| power| 168.0  | W    | 2014-08-14T07:24:03.808000 |
| node-41.domain.tld| power| 174.0  | W    | 2014-08-14T07:23:53.691000 |
| node-41.domain.tld| power| 169.0  | W    | 2014-08-14T07:23:43.585000 |
+-------------------+------+--------+------+----------------------------+

The energy consumed, measured in Watts, is in the Volume field and there is a timestamp which indicates when the sample was taken by Kwapi.

Et voila – you’ve now set up Kwapi with Ceilometer and you can monitor the energy consumption of your Openstack cluster.

If you’re interested in this topic, you may be interested in some of our other blog posts:

• A Web Application to Monitor and Understand Energy Consumption in an Openstack Cloud
• Understanding the relationship between ceilometer processor utilisation and system energy consumption for a basic scenario in Openstack
• Migration of Ceilometer Energy Consumption Data from Havana/MySQL to Icehouse/MongoDB