Following our previous blog post, we are still looking at tools for collecting metrics from an Openstack deployment in order to understand its resource utilization. Although Monasca has a comprehensive set of metrics and alarm definitions, the complex installation process combined with a lack of documentation makes it a frustrating experience to get it up and running. Further, although it is complex, with many moving parts, it was difficult to configure it to obtain the analysis we wanted from the raw data, viz how many of our servers are overloaded over different timescales in different respects (cpu, memory, disk io, network io). For these reasons we decided to try Prometheus with Grafana which turned out to be much easier to install and configure (taking less than an hour to set up!). This blog post covers the installation process and configuration of Prometheus and Grafana in a Docker container and how to install and configure Canonical’s Prometheus Openstack exporter to collect a small set of metrics related to an Openstack deployment.

Note that minor changes to this HOWTO are required to install theses services in a VM or in a host machine when using containers is not an option. As preparation, take note of your Openstack deployment’s locations for Keystone and the Docker host. Remember that all downloads should be verified by signature comparison for production use.

Installing and configuring Prometheus

First of all pull the Ubuntu image into you docker machine. Let’s call it docker-host.

Note that in this blog post we describe Prometheus installation process step-by-step – we chose to install it from scratch to get a better understanding of the system, but using the pre-canned Docker Hub image is also possible.

docker pull ubuntu:14.04

Then create the docker container opening the port 9090 which will be used to get/push metrics into Prometheus.

docker run -it -p 9090:9090 --name prometheus ubuntu:14.04

Inside the container download the latest version of Prometheus and uncompress it (version 1.3.1 is used in this HOWTO; the download size is ca. 16 MB).

wget https://github.com/prometheus/prometheus/releases/download/v1.3.1/prometheus-1.3.1.linux-amd64.tar.gz
tar xvf prometheus-1.3.1.linux-amd64.tar.gz
cd prometheus-1.3.1.linux-amd64

Configure prometheus.yml adding the targets from which prometheus should scrape metrics. See the example below for the Openstack exporter (assuming it is installed in the same docker-host):

scrape_configs:
  - job_name: 'openstack-deployment-1'
    scrape_interval: 5m
    Static_configs:
      - targets: ['docker-host:9183']

Start the Prometheus service:

./prometheus -config.file=prometheus.yml

Similarly, install and configure the Prometheus Openstack exporter in another container. Note that this container needs to be set up manually as there are configuration files to be changed and Openstack libraries to be installed.

docker run -it -p 9183:9183 --name prometheus-openstack-exporter ubuntu:14.04
sudo apt-get install python-neutronclient python-novaclient python-keystoneclient python-netaddr unzip wget python-pip python-dev python-yaml
pip install prometheus_client
wget https://github.com/CanonicalLtd/prometheus-openstack-exporter/archive/master.zip
unzip master.zip
cd prometheus-openstack-exporter-master/

Next, configure prometheus-openstack-exporter.yaml create the /var/cache/prometheus-openstack-exporter/ directory and the novarc file containing credentials for Nova user.

mkdir /var/cache/prometheus-openstack-exporter/
echo  export OS_USERNAME=nova-username \
      export OS_PASSWORD=nova-password \
      export OS_AUTH_URL=http://keystone-url:5000/v2.0 \
      export OS_REGION_NAME=RegionOne \
      export OS_TENANT_NAME=services > novarc
source novarc
./prometheus-openstack-exporter prometheus-openstack-exporter.yaml

Then you’ve got a fully functional Prometheus system with some Openstack metrics on it! Visit http://docker-host:9090 to graph and see which metrics are available.

Here is the list of the 18 metrics currently collected by Prometheus Openstack exporter:

Alternatively you could use Prometheus’s Node exporter for more detailed metrics on node usage – this needs to be installed in the controller/compute nodes and the prometheus.yml configuration file also needs to be changed. A docker container is also available at Docker Hub.

Although Prometheus provides some rudimentary graph support, combining it with a more powerful graphing solution makes it much easier to see what’s going on in your system. For this reason, we set up Grafana.

Installing Grafana

The latest version of Grafana (currently 4.0.0-beta2) had a lot of improvements in its user interface, it also supports now alerting and notifications for every panel available – refer to the documentation for more information. Its integration with Prometheus is very straightforward, as described below.

First of all, pull the grafana image into your docker-host and create the docker container opening the port 3000 used to access it.

docker pull grafana/grafana
docker run -d -p 3000:3000 grafana/grafana:4.0.0-beta2

Visit http://docker-host:3000 and use the credentials admin/admin to log into the dashboard. In the Data Sources tab, add a new corresponding data source.

Create a new dashboard and add panels containing graphs using the Prometheus datasource.

Play around with metrics available and create your own dashboard! See a simple example below.

Conclusion

Although not many metrics are available yet to monitor an Openstack deployment the combination of Prometheus and Grafana is quite powerful for visualising data; also it was much easier to set up in comparison with Monasca. Further, from a cursory glance, Prometheus seems to be more flexible than Monasca and for these reasons it appears more promising. That said, we are still looking into Prometheus and how it can be used to properly understand resource consumption in an Openstack context, but that will come in another blog post!