Installing Monasca – a happy ending after much sadness and woe

In one of our projects we are making contributions to an Openstack project called Watcher, this project focuses on optimizing resource utilization of a cloud according to a given strategy. As part of this work it is important to understand the resource utilization of the cloud beforehand in order to make a meaningful contribution. This requires collection of metrics from the system and processing them to understand how the system is performing. The Ceilometer project was our default choice for collecting metrics in an Openstack deployment but as work has evolved we are also exploring alternatives – specifically Monasca. In this blog post I will cover my personal experience installing Monasca (which was more challenging than expected) and how we hacked the monasca/demo docker image to connect it to our Openstack deployment.

In our earlier series of blog posts we explained how to install, configure and test Monasca step-by-step using git, Maven and pip – a good option for learning the moving parts of the giant beast that is Monasca and how it is all linked together. This content was posted over a year ago and hence it is now a little bit stale; since that time work has been done on deployment using automation tools such as  Ansible or Puppet. Generally, when working on this, we encountered significant issues with lack of documentation, unspecified versioning issues, unclear dependencies etc.

Our first attempt to install Monasca was based on different Ansible repositories available on Github – pointed to by the official page of Monasca. Although most of the repositories are Ansible roles, some try to put everything in a single place. Interestingly, many of those assume that Openstack is deployed with Devstack (which was not true in our case – we had a small  Packstack deployment) and this caused many steps to fail during the installation as files were not where they were expected to be (in devstack many files are into /opt/stack/). Even after solving these issues Monasca itself did not seem to work and we always ended up with a system where only few monasca services were up and running. We’re not certain where the problems were, but there were multiple issues with Kafka and in some cases it was not possible to get monasca-agent or monasca-notifications working.

A tip if you want to go this path: Most of the repos use Ansible 1.9.2, which is not obvious. We had many issues using Ansible >= 2.0 and some roles using version 1.9.2 which required modification to the source code as there were changes in how loops are defined.

After the frustrating experience with Ansible we then decided to get back to basics, there is a functional out-of-the-box docker image in Docker Hub used for testing Monasca. It contains pretty much everything Monasca needs for running – even a local Keystone, Nova and Glance. We then decided to modify Monasca components in this image in order to connect it to our to Openstack cluster. Note that only Openstack credentials were modified inside the container – InfluxDB, MySQL and other services credentials also provided in the container were not modified. After relatively little tinkering, we managed to get this working as described below.

First of all, pull the image into your docker machine – let’s call it docker-host. Note that docker-host must be able to access your Openstack deployment (via the public APIs) and vice-versa

docker pull monasca/demo

Next, create the docker container but it is necessary to modify its entrypoint, so it does not start setting up monasca automatically and we can change the Monasca configuration files; also, note that the since it will not be running Keystone, we will remove the Keystone port:

docker run -it -p {port_to_horizon}:80 -p 8080:8080 --name monasca --entrypoint=/bin/bash monasca/demo

Change the value of port_to_horizon in case port 80 is already used in docker-host.

Inside the docker container you will be in the ‘/setup’ directory, in this directory you will see the  ansible files used to set up this container. The main file in this directory is called demo-start.sh, this file will start all the services required by Monasca but before running this script it is necessary to modify the following Monasca Openstack configuration details.

In /etc/monasca/api-config.yml add your credentials in the middleware section (note that you will need to install a text editor in the container):

##  /etc/monasca/api-config.yml
# region usually defaults to RegionOne
region: “{SERVICE_REGION}”
...
serverVIP: "KEYSTONE_URL"
serverPort: KEYSTONE_PORT
defaultAuthorizedRoles: [user, domainuser, domainadmin, monasca-user, admin]
agentAuthorizedRoles: [monasca-agent]
adminAuthMethod: password
adminUser: "monasca"
adminPassword: "MONASCA_PASSWORD"
adminProjectName: "monasca"

Similarly in /usr/local/bin/monasca-reconfigure and /setup/alarms.yml:

##  /usr/local/bin/monasca-reconfigure
#!/bin/sh
'/opt/monasca/bin/monasca-setup' \
     -u 'monasca-agent' \
     -p 'MONASCA_AGENT_PASSWORD' \
      -s 'monitoring'  \
     --keystone_url 'KEYSTONE_URL' \
     --project_name 'MONASCA_AGENT_PROJECT' \
       --monasca_url 'http://localhost:8080/v2.0'  \
      \
      --check_frequency '5'  \
      \
      \
     --overwrite

## /setup/alarms.yml
- name: Setup default alarms
   hosts: localhost
   vars:
       keystone_url: KEYSTONE_URL
       keystone_user: monasca
       keystone_password: MONASCA_PASSWORD
       keystone_project: monasca
   roles:
       - {role: monasca-default-alarms, tags: [alarms]}

In /setup/demo-start.sh there is an error regarding the location of monasca-notification init script, modify the ‘/usr/local/bin/monasca-notification &’ (line 44) to ‘/opt/monasca/bin/monasca-notification &’. Also remove the playbook which installs the Keystone service (line 11):

## /setup/demo-start.sh
…
# remove this line 
# ansible-playbook -i /setup/hosts /setup/keystone.yml -c local
            ...
/opt/monasca/bin/monasca-notification &

Finally remove services which are not necessary anymore, eg the local Keystone, from start.sh, leave apache2 and change the variable OPENSTACK_HOST in /etc/openstack-dashboard/local_settings.py to the IP where Keystone is installed:

## /setup/start.sh
#!/bin/bash
/etc/init.d/apache2 start

## /etc/openstack-dashboard/local_setting.py
OPENSTACK_HOST = "KEYSTONE_IP"

Before running the main script to set everything up make sure you have created the Keystone users, projects and roles which are assumed in the way monasca is configured:

 keystone tenant-create --name monasca --description "Monasca tenant"
 keystone user-create --name monasca-agent --pass password --tenant [monasca-tenant-id]
 keystone user-create --name monasca --pass password --tenant [monasca-tenant-id]
 keystone role-create --name monasca-agent
 keystone role-create --name monasca-user
 keystone user-role-add --user [monasca-agent-id] --role [monasca-agent-role-id] --tenant [monasca-tenant-id]
 keystone user-role-add --user [monasca-id] --role [monasca-user-role-id] --tenant [monasca-tenant-id]
 keystone service-create --type monitoring --name monasca --description "Monasca monitoring service"
 keystone endpoint-create --service [service-id] --publicurl http://docker-host:8080/v2.0 --internalurl http://docker-host:8080/v2.0 --adminurl http://docker-host:8080/v2.0

Once all of this is done, you can now run /setup/demo-start.sh to set everything up – it may take a couple of minutes. When it has finished, you can visit your monasca-dashboard page at http://docker-host:{port_to_horizon}. Log in as the user Monasca and visit the monitoring tab in the dashboard.

In this tab you will be able to create alarms definitions and notifications. You will also be able to see an overview of the current state of the system in the Grafana dashboard. To collect more data regarding your Openstack cluster you will need to install and configure monasca-agent service in each of the controller/compute nodes. It is possible to do it in a virtualenv so it does not modify any library in the system.

# on compute/controller nodes
virtualenv monasca_agent_env
source monasca_agent_env/bin/activate
pip install --upgrade monasca-agent

monasca-setup -u monasca-agent -p password --project_name monasca -s monitoring --keystone_url {KEYSTONE_URL} --monasca_url http://docker-host:8080/v2.0 --config_dir /etc/monasca/agent --log_dir /var/log/monasca/agent --overwrite

Then you’ve got a fully functional Monasca system which works with your Openstack cluster – it’s a good solution for experimenting with Monasca to understand how powerful it really is. This solution is probably not the best solution for a production deployment, however – it would probably make more sense to decouple the services and put the data in a persistent data store, but that’s another day’s work!

Troubleshooting

If you are getting “requests.exceptions.ConnectionError: (‘Connection aborted.’, gaierror(-2, ‘Name or service not known’))” error when running demo-start.sh script that means the container is not able reach the endpoint defined in Openstack for the monitoring service. Make sure that the container is able to connect to the external interface in your Openstack cluster and the Keystone endpoint for the monitoring service is configured correctly.

The demo-start.sh is the main script in this container, in case you find any issues you can execute each of the commands in this script sequentially to understand more clearly what is happenning.  

“Exception in thread “main” java.lang.RuntimeException: Topology with name `thresh-cluster` already exists on cluster”. This error is known to happen often as the thresh-cluster service tries to configure itself for a second time, you can ignore it.