Cyclops 2.0 supports Ceilometer Data Collection!

As announced in our last blogpost about the official release of Cyclops 2.0, which is finally out and is adding new features.

The collector that is being released today is the Ceilometer Usage Collector. This collector enables Cyclops 2.0 to provide full rating, charging and billing support to an OpenStack deployment using the data provided by Ceilometer.

In addition to the announced features, our team has pushed forward in the development of the new Usage Collectors. The Usage Collectors are the entry point of data for the Framework itself. They consist of isolated microservices that gather data from a specific provider and distributes it via RabbitMQ to the UDR microservice.

Screen Shot 2016-07-11 at 16 Continue reading

Managing ceilometer data in openstack

Ceilometer can collect a large amount of data, particularly in a system with large amount of servers and high activity: in such a scenario, the numbers of meters and samples can be large which  affects ceilometer performance and gives rise to quite large databases. In our particular case we are studying energy consumption in servers and how resource utilization (mainly cpu) may relates to overall energy consumption. The energy data is collected through Kwapi and stored in ceilometer every 10 seconds (yes, this is probably too fine-grained!). We had problems that the database accumulated too quickly, filling up the root disk partition on the controller and causing significant problems for the system. In this blog post, we describe the approach we now use for managing ceilometer data which ensures that the resources consumed by ceilometer remain under control. Continue reading

Using Ceilometer Data to Determine which VMs ran on a given Physical host in Openstack

In our previous blog post we showed a web application to monitor and understand energy consumption in an Openstack cluster; the main goal of this tool is to understand how energy consumption relates to activity on the cloud resources. In general, this can be quite complex as there are many resources to take in account and Ceilometer doesn’t report all the information required at this time. In this blog post we describe one task we needed to perform in this work: we needed to retrieve from Ceilometer the set of VMs running on a given physical server together with their CPU utilization over some period of time. We can envisage other contexts in which it might be useful to do this, so we present the solution here. (Note that it is of course possible to obtain the set of VMs that are currently running on a given physical host using Nova, but this does not offer the historical perspective).

Continue reading

A Web Application to Monitor and Understand Energy Consumption in an Openstack Cloud

In one of our projects we need to understand the energy consumption of our servers. Our initial work in this direction involved collecting energy consumption data using Kwapi and storing it in Ceilometer for further study. The data stored in Ceilometer is valuable; however, it is insufficient to really understand energy consumption in detail. Consequently, we are developing a web application which gives a much greater insight into energy consumption in our cloud resources. This is very much a work in progress, so this post just highlights a few points relating to the application as well as a video which shows the current version of the application.

The tool was developed to be totally integrated with Openstack. Users log in with their Openstack credentials (using Keystone authentication) and are  redirected to the overview page where they can see  the total energy consumed by the VMs in their projects for the the previous month as well as some  general information regarding virtual machines; a line chart displays how energy consumed varies over time.

Continue reading

Understanding the relationship between ceilometer processor utilisation and system energy consumption for a basic scenario in Openstack

In one of our earlier blog posts, we described some test we performed to determine how server power consumption increases with compute load; this post is something of a variation on that post, but here we put the focus on work taking place within VMs rather than work taking place within the host OS. The point here is to understand how VM load and energy consumption correlate. Here we document the results obtained.

As with our previous work, we focused on compute bound loads – the focus in this test is on increasing the compute load on the servers by performing π calculations inside the VM. In this work, we used homogeneous VMs – all the VMs were of the same flavor with the following configuration 2GB RAM, 20GB local disk and 1 VCPU.

Continue reading

Collecting energy consumption data using Kwapi in Openstack

In some of our projects we need to understand the energy consumption of our servers in an Openstack cluster. The first step in this process was to collect energy consumption data from our IBM servers; we stored this in Ceilometer for further study. In this blog post we will cover how we do this.

First, Kwapi 101.

Kwapi is a part of the openstack ecosystem (perhaps a little peripheral) which is focused on collecting energy data. It has pretty good integration with Ceilometer which enables the energy data to be stored there.

Kwapi is architected in such a way that individual drivers listen to a specific wattmeter – a wattmeter can be a physical energy meter with a wifi interface or connected to ipmi, i.e. any kind of device that measures energy consumption. The drivers then pass the information on to plug-ins. These are the API, the forwarder and the RRD plugin which is the visualization plugin that provides a web interface with power consumption graphs.

Continue reading

Profiling the Ceilometer API to Identify Performance Bottlenecks

We are using ceilometer to collect data energy from our servers. As noted previously we were having some performance issues and we needed to investigate further. In this blog post we will cover our approach to performing profiling on ceilometer API to determine where the problems arose.

Of course, the first step was to take a look at the log files (in /var/log/ceilometer-all.log); as there was nothing unusual in there, we decided to perform profiling of the code.

Continue reading

Advanced Queries to Ceilometer with a mongo backend

We are changing our ceilometer backend from mysql to mongodb in one of our experimental Openstack installations. The reason for this change is that mongo seems to deliver better ceilometer performance than mysql; further ceilometer data structures are a more natural fit with mongo (and indeed, this is largely where they came from). This can be seen in a typical record below which is clearly hierarchical and contains so-called embedded documents (in mongo terminology):

"_id" : ObjectId("53bbe7ea926fc4597b42aafc"),
"counter_name" : "instance",
"resource_metadata" : {
    "status" : "active",
    "display_name" : "Test",
    "name" : "instance-00000001",
    "image" : {
        "id" : "bdfaab74-6542-4cbb-94f1-5306662208a7",
        "name" : "cirros-0.3.2-x86_64-uec"
    },
    "host" : "7c261f3a33c099538d448be797e5ce0c0d8cf8bf9f75dd59ce04df86",
}
...

Mongo natively provides support for queries of data structured in this fashion. More specifically, mongo enables data at different levels of the hierarchy to be queried – something which is difficult in SQL.

In python, this can be done simply as follows:


query = [{'field': 'timestamp', 'op': 'gt', 'value': date},
{'field': 'metadata.status', 'op': 'eq', 'value': 'active'}]

sample_list = ceilometer.samples.list(meter_name='instance', q=query)

The interesting point, which we did not understand clearly until now, is that ceilometer with a mongo backend supports exactly this type of query. Thus, the following command line query can obtain all instances that were active for a certain period of time:


ceilometer sample-list -m instance -q “timestamp>date; metadata.status=’active’” 

Then ceilometer will return all the samples of instances active in this time range.

+--------------------------------------+-----------+-----------+--------------------+
| Resource ID                          | Name      | Type     | Timestamp           |
+--------------------------------------------------+----------+---------------------+
| 7535b9f6-01a6-410e-980d-338031e7a2c4 | instance  | instance | 2014-07-09T09:30:05 |
| 7535b9f6-01a6-410e-980d-338031e7a2c4 | instance  | instance | 2014-07-09T09:20:05 |
| 7535b9f6-01a6-410e-980d-338031e7a2c4 | instance  | instance | 2014-07-09T09:10:04 |
+--------------------------------------+-----------+----------+---------------------+

Querying ceilometer with a mysql database in this fashion results in an error.

Full-stack Monitoring for OpenStack

Introduction

The Ceilometer project is a core OpenStack service that provide collection of metering data on managed virtual resources (e.g. compute, storage, networking). Before it was only possible to collect data from OpenStack virtual resources which are running upon an OpenStack deployment. The work presented hereafter addresses this issue outlines a means to monitor both virtual and physical resources in an integrated common approach. This work has been appearsd in the upcoming OpenStack release named “Icehouse”.

Continue reading