Installing Monasca – a happy ending after much sadness and woe

In one of our projects we are making contributions to an Openstack project called Watcher, this project focuses on optimizing resource utilization of a cloud according to a given strategy. As part of this work it is important to understand the resource utilization of the cloud beforehand in order to make a meaningful contribution. This requires collection of metrics from the system and processing them to understand how the system is performing. The Ceilometer project was our default choice for collecting metrics in an Openstack deployment but as work has evolved we are also exploring alternatives – specifically Monasca. In this blog post I will cover my personal experience installing Monasca (which was more challenging than expected) and how we hacked the monasca/demo docker image to connect it to our Openstack deployment.

In our earlier series of blog posts we explained how to install, configure and test Monasca step-by-step using git, Maven and pip – a good option for learning the moving parts of the giant beast that is Monasca and how it is all linked together. This content was posted over a year ago and hence it is now a little bit stale; since that time work has been done on deployment using automation tools such as  Ansible or Puppet. Generally, when working on this, we encountered significant issues with lack of documentation, unspecified versioning issues, unclear dependencies etc.

Our first attempt to install Monasca was based on different Ansible repositories available on Github – pointed to by the official page of Monasca. Although most of the repositories are Ansible roles, some try to put everything in a single place. Interestingly, many of those assume that Openstack is deployed with Devstack (which was not true in our case – we had a small  Packstack deployment) and this caused many steps to fail during the installation as files were not where they were expected to be (in devstack many files are into /opt/stack/). Even after solving these issues Monasca itself did not seem to work and we always ended up with a system where only few monasca services were up and running. We’re not certain where the problems were, but there were multiple issues with Kafka and in some cases it was not possible to get monasca-agent or monasca-notifications working.

A tip if you want to go this path: Most of the repos use Ansible 1.9.2, which is not obvious. We had many issues using Ansible >= 2.0 and some roles using version 1.9.2 which required modification to the source code as there were changes in how loops are defined.

After the frustrating experience with Ansible we then decided to get back to basics, there is a functional out-of-the-box docker image in Docker Hub used for testing Monasca. It contains pretty much everything Monasca needs for running – even a local Keystone, Nova and Glance. We then decided to modify Monasca components in this image in order to connect it to our to Openstack cluster. Note that only Openstack credentials were modified inside the container – InfluxDB, MySQL and other services credentials also provided in the container were not modified. After relatively little tinkering, we managed to get this working as described below.

First of all, pull the image into your docker machine – let’s call it docker-host. Note that docker-host must be able to access your Openstack deployment (via the public APIs) and vice-versa


docker pull monasca/demo

Next, create the docker container but it is necessary to modify its entrypoint, so it does not start setting up monasca automatically and we can change the Monasca configuration files; also, note that the since it will not be running Keystone, we will remove the Keystone port:


docker run -it -p {port_to_horizon}:80 -p 8080:8080 --name monasca --entrypoint=/bin/bash monasca/demo

Change the value of port_to_horizon in case port 80 is already used in docker-host.

Inside the docker container you will be in the ‘/setup’ directory, in this directory you will see the  ansible files used to set up this container. The main file in this directory is called demo-start.sh, this file will start all the services required by Monasca but before running this script it is necessary to modify the following Monasca Openstack configuration details.

In /etc/monasca/api-config.yml add your credentials in the middleware section (note that you will need to install a text editor in the container):


##  /etc/monasca/api-config.yml
# region usually defaults to RegionOne
region: “{SERVICE_REGION}”
...
serverVIP: "KEYSTONE_URL"
serverPort: KEYSTONE_PORT
defaultAuthorizedRoles: [user, domainuser, domainadmin, monasca-user, admin]
agentAuthorizedRoles: [monasca-agent]
adminAuthMethod: password
adminUser: "monasca"
adminPassword: "MONASCA_PASSWORD"
adminProjectName: "monasca"

Similarly in /usr/local/bin/monasca-reconfigure and /setup/alarms.yml:


##  /usr/local/bin/monasca-reconfigure
#!/bin/sh
'/opt/monasca/bin/monasca-setup' \
     -u 'monasca-agent' \
     -p 'MONASCA_AGENT_PASSWORD' \
      -s 'monitoring'  \
     --keystone_url 'KEYSTONE_URL' \
     --project_name 'MONASCA_AGENT_PROJECT' \
       --monasca_url 'http://localhost:8080/v2.0'  \
      \
      --check_frequency '5'  \
      \
      \
     --overwrite

## /setup/alarms.yml
- name: Setup default alarms
   hosts: localhost
   vars:
       keystone_url: KEYSTONE_URL
       keystone_user: monasca
       keystone_password: MONASCA_PASSWORD
       keystone_project: monasca
   roles:
       - {role: monasca-default-alarms, tags: [alarms]}

In /setup/demo-start.sh there is an error regarding the location of monasca-notification init script, modify the ‘/usr/local/bin/monasca-notification &’ (line 44) to ‘/opt/monasca/bin/monasca-notification &’. Also remove the playbook which installs the Keystone service (line 11):


## /setup/demo-start.sh

# remove this line 
# ansible-playbook -i /setup/hosts /setup/keystone.yml -c local
            ...
/opt/monasca/bin/monasca-notification &

Finally remove services which are not necessary anymore, eg the local Keystone, from start.sh, leave apache2 and change the variable OPENSTACK_HOST in /etc/openstack-dashboard/local_settings.py to the IP where Keystone is installed:


## /setup/start.sh
#!/bin/bash
/etc/init.d/apache2 start

## /etc/openstack-dashboard/local_setting.py
OPENSTACK_HOST = "KEYSTONE_IP"

Before running the main script to set everything up make sure you have created the Keystone users, projects and roles which are assumed in the way monasca is configured:


 keystone tenant-create --name monasca --description "Monasca tenant"
 keystone user-create --name monasca-agent --pass password --tenant [monasca-tenant-id]
 keystone user-create --name monasca --pass password --tenant [monasca-tenant-id]
 keystone role-create --name monasca-agent
 keystone role-create --name monasca-user
 keystone user-role-add --user [monasca-agent-id] --role [monasca-agent-role-id] --tenant [monasca-tenant-id]
 keystone user-role-add --user [monasca-id] --role [monasca-user-role-id] --tenant [monasca-tenant-id]
 keystone service-create --type monitoring --name monasca --description "Monasca monitoring service"
 keystone endpoint-create --service [service-id] --publicurl http://docker-host:8080/v2.0 --internalurl http://docker-host:8080/v2.0 --adminurl http://docker-host:8080/v2.0

Once all of this is done, you can now run /setup/demo-start.sh to set everything up – it may take a couple of minutes. When it has finished, you can visit your monasca-dashboard page at http://docker-host:{port_to_horizon}. Log in as the user Monasca and visit the monitoring tab in the dashboard.

In this tab you will be able to create alarms definitions and notifications. You will also be able to see an overview of the current state of the system in the Grafana dashboard. To collect more data regarding your Openstack cluster you will need to install and configure monasca-agent service in each of the controller/compute nodes. It is possible to do it in a virtualenv so it does not modify any library in the system.


# on compute/controller nodes
virtualenv monasca_agent_env
source monasca_agent_env/bin/activate
pip install --upgrade monasca-agent

monasca-setup -u monasca-agent -p password --project_name monasca -s monitoring --keystone_url {KEYSTONE_URL} --monasca_url http://docker-host:8080/v2.0 --config_dir /etc/monasca/agent --log_dir /var/log/monasca/agent --overwrite

Then you’ve got a fully functional Monasca system which works with your Openstack cluster – it’s a good solution for experimenting with Monasca to understand how powerful it really is. This solution is probably not the best solution for a production deployment, however – it would probably make more sense to decouple the services and put the data in a persistent data store, but that’s another day’s work!

Troubleshooting

If you are getting “requests.exceptions.ConnectionError: (‘Connection aborted.’, gaierror(-2, ‘Name or service not known’))” error when running demo-start.sh script that means the container is not able reach the endpoint defined in Openstack for the monitoring service. Make sure that the container is able to connect to the external interface in your Openstack cluster and the Keystone endpoint for the monitoring service is configured correctly.

The demo-start.sh is the main script in this container, in case you find any issues you can execute each of the commands in this script sequentially to understand more clearly what is happenning.  

“Exception in thread “main” java.lang.RuntimeException: Topology with name `thresh-cluster` already exists on cluster”. This error is known to happen often as the thresh-cluster service tries to configure itself for a second time, you can ignore it.


16 Kommentare

  • I’ve met the following problem:
    Could not find domain: default

    It comes when I try to setup alarms with the next command:
    ansible-playbook -i /setup/hosts /setup/alarms.yml -c local

    I also cannot execute “monasca metric-list” because of the same problem.

    Have you met something like that? I have openstack newton.

    • Hi Alexander,

      Which version of Keystone are you using?
      I haven’t seen this error setting up the container but my guess is that you are using Keystone v2.0 instead of V3 when configuring KEYSTONE_URL in api-config.yml, alarms.yml etc.
      If you have more details of this error please send an email to gaea@zhaw.ch and I will have a look.

        • This seems to be an issue with the user ‘monasca’ or ‘monasca-agent’ not being able to authenticate against Keystone.
          Check if the domain id ‘default’ is in Keystone and if the users mentioned above are able to create a Keystone token and use it in other Openstack services.

          Also make sure that the container and the controller node are able to access the endpoint listed for the monitoring service in Keystone.

  • when i run command:
    monasca-setup -u monasca-agent -p password –project_name monasca -s monitoring –keystone_url {KEYSTONE_URL} –monasca_url http://docker-host:8080/v2.0 –config_dir /etc/monasca/agent –log_dir /var/log/monasca/agent –overwrite

    I get the issue, i installed oslo.utils 3.20.0 package python and “service monasca-agent start”, get error:

    pkg_resources.VersionConflict: (oslo.utils 3.20.0 (/home/tmp/monasca_agent_env/lib/python2.7/site-packages), Requirement.parse(‘oslo.utils<=3.16.0'))

    So, i reinstall oslo.utils to version =3.18.0’)). 😐

    How to fix it?

    • So, i reinstall oslo.utils to version =3.18.0’)).
      Monasca-agent require downgrade to version 3.16.0. Then start monasca-agent, error again that required upgrade to version 3.18.0 :|..

      It seem like loop. @@
      how to fix it?

    • It seems that there is a dependency issue in the lastest version of monasca-agent.
      We used monasca-agent version 1.1.24, you can install it using:
      pip install –upgrade monasca-agent==1.1.24
      This requires oslo.utils==3.18.0.
      Let us know if this solution worked for you.

      • Yup, It worked. And i want to modify running port of supervisord because i get error:

        Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting supervisord.

        I found in /etc/monasca/agent/supervisor.conf but no line in order to modify running port of it.

        I found this line in /etc/monasca/agent/agent.yaml:
        “monasca_statsd_port: 8125” but no effect.

        • I set up another VM and nothing running on it. So, i still get error like this:

          Starting Monasca Monitoring Agent (using supervisord) monasca-agent
          Error: Another program is already listening on a port that one of our HTTP servers is configured to use. Shut this program down first before starting supervisord.
          For help, use /usr/local/bin/supervisord -h

          • I solved this issue by delete /var/tmp/monasca-agent-supervisor.sock and start service monasca-agent. It’s running.

            Thank your help!

  • When created the Keystone users, projects and roles on Openstack. May be you miss defined RegionOne, if miss parameter –region RegionOne, it’ll create another region “regionOne” and monasca-agent doesn’t work. When set to RegionOne for service monitoring, monasca-agent worked. 🙂

  • when i start monasca-agent, i get error:
    monasca_agent.forwarder.api.monasca_api(monasca_api.py:167) | Invalid token detected. Waiting to get new token from Keystone. Queuing the messages to send later…

    Any solution to fix? I use Keystone port 35357 with v3.

    • Did you try using port 5000 with v3?
      There must be a parameter missing in your monasca-agent config file.
      We had issues before with user_domain_id and project_domain_id.

  • we have installed Openstack w/ Monasca. when i login and open the dashboard page, I dont see any “Openstack Services” show up in the Monitoring tab. I expected the “Services” and the “Servers” monitored by monasca to show up on the Monitoring tab. What might be the issue here? Please assist.

  • I am following your instructions to configure the Monasca Docker to work with DevStack Pike.

    I can access the Monasca UI login page in the web browser, but I am not able to log in. What user name and password should I be using, i.e. in which file are they defined?

    Thank you.

    • Hi George,

      You will be able to find this information directly in keystone. There is a section in this blog post where you create a user for monasca, here a sample from a command line listed there:
      keystone user-create --name monasca --pass password --tenant [monasca-tenant-id]
      You could use that user to login in web browser.


Leave a Reply

Your email address will not be published. Required fields are marked *