{"id":7001,"date":"2014-12-12T18:00:03","date_gmt":"2014-12-12T16:00:03","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=7001"},"modified":"2014-12-16T10:29:37","modified_gmt":"2014-12-16T08:29:37","slug":"managing-ceilometer-data-in-openstack","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/managing-ceilometer-data-in-openstack\/","title":{"rendered":"Managing ceilometer data in openstack"},"content":{"rendered":"

Ceilometer can collect a large amount of data, particularly in a system with large amount of servers and high activity: in such a scenario, the numbers of meters and samples can be large which \u00a0affects ceilometer performance and gives rise to quite large databases. In our particular case we are studying energy consumption in servers and how resource utilization (mainly cpu) may relates to overall energy consumption. The energy data is collected through Kwapi and stored in ceilometer every 10 seconds (yes, this is probably too fine-grained!). We had problems that the database accumulated too quickly, filling up the root disk partition on the controller and causing significant problems for the system. In this blog post, we describe the approach we now use for managing ceilometer data which ensures that the resources consumed by ceilometer remain under control.<\/p>\n

In some cases, if you have done a standalone installation and haven\u2019t considered ceilometer at the installation stage, the database may be \u00a0configured in the root partition: this is not ideal for quite obvious reasons – filling up the root file system can render the system inoperable. \u00a0For this reason, we advise that the ceilometer db backend be put on a partition different from the root partition. For mongodb, this is as simple as changing the dbpath parameter in the \/etc\/mongodb.conf config file. Then a simple restart of mongo is required.<\/p>\n

Backing up Mongodb Data<\/b><\/h2>\n

Now with the database up and running in a different partition we can trigger a cron job which backs up the data from ceilometer regularly. We wrote a simple python script to dump selected contents of the mongodb monthly and compress it – the data can then be archived somewhere else. In our case, we found that the dumped data compresses down to approximately 10% of the size of the dump.<\/p>\n

The simple python script<\/a> we wrote uses mongodump inside the python subprocess library<\/a> (it only works with Mongodb<\/a> databases right now). Subprocess was used because python mongodb libraries such as pymongo are not really intended for large import or exports – they are more intended for standard database interactions. \u00a0We did, however, need to use pymongo to understand which collections exist in the db and this information was then used as an input to mongodump. In the script, the collections are then backed up in a temporary directory using zip.<\/p>\n

The script requires <\/i>pymongo<\/i><\/a> and zip support as well as ZIP64_SUPPORT for files larger than 64GB. The tmp and zip directories can be changed in the script.<\/i><\/p>\n

Setting up ceilometer-expirer<\/b><\/h2>\n

Setting up an archiving system as above, is not sufficient to ensure that the db size is controlled – it is also necessary to remove redundant data from the db. Fortunately, with mongo this is very straightforward: mongo has support for expiry of data and performs internal cleanup operations to remove expired data.<\/p>\n

Ceilometer has support for data expiry and the time_to_live parameter in ceilometer.conf controls how long the data will remain in the database. In general, the time_to_live for samples defaults \u00a0to -1 in ceilometer, which means that the samples have no expiry date. The ceilometer-expirer component of ceilometer manages data expiry and its behaviour is dependent on which db backend is being used<\/a>. In the case of mongo, if the time_to_live parameter is not -1, a new index is added to collections which has a timestamp field and mongo deletes the samples older than the configured ttl value. In the case of other dbs, such as mysql, ceilometer-expirer proactively removes entries from the database which have expired.<\/p>\n

It is worth noting that as well as the expired meter\/sample data, Ceilometer-expirer removes other entries in the db which which are not linked to any other entries, e.g. information relating to an instance which was terminated some time ago.<\/p>\n

In case you want to add expiry information to a particular collection directly within the mongo backend it is possible using the following command in the mongodb client:<\/p>\n

db.meter.ensureIndex( { “timestamp”: 1 }, { expireAfterSeconds: 2764800 } )<\/p>\n

However, using this approach is not advised – it is much more sensible to configure ceilometer appropriately to ensure data expires correctly.<\/p>\n

The above basic approach to ceilometer data management should ensure that ceilometer data does not grow uncontrollably and makes it highly unlikely that ceilometer data would render the system inoperable.<\/p>\n

If you’re interested in this topic, you may be interested in some of our other blog posts:<\/p>\n