How to install and setup Monasca (3/3)

In this final part of the tutorial we will verify that all things are working properly.

Because of the large number of moving components the norm is that things will go wrong. Code changes daily both in Monasca and in Openstack and quite often things need further adjusting.

As a general rule, in case a component is misbehaving, change logging from INFO to DEBUG in /etc/monasca and restart the component. The look at the logs in /var/log/monasca and see what is wrong. For Thresh / Storm, logs can be found in /opt/storm/current/logs

In order to test if monasca-ui is properly configured log into Horizon with

username: monasca
password: password

And the monitoring panel should be visible. In case it isn’t, monasca-ui might not be getting what it expects from Keystone.

In order to test monasca-api, first thing you need to do is get a valid keystone toke (let’s say for admin). Then, on a machine which has the monasca client installed run:

export OS_USERNAME=monasca
export OS_PASSWORD=password
export OS_AUTH_URL=http://devstack:5000/v3
monasca alarm-definition-list

That should return nothing as no definitions are present. Next look at some of the metrics which are being collected.

ubuntu@devstack:~$ monasca metric-list
+-----------------------------+-------------------------------------------+
| name                        | dimensions                                |
+-----------------------------+-------------------------------------------+
| cpu.idle_perc               | hostname: monasca                         |
|                             | service: monitoring                       |
| cpu.stolen_perc             | hostname: monasca                         |
|                             | service: monitoring                       |
| cpu.system_perc             | hostname: monasca                         |
|                             | service: monitoring                       |
| cpu.user_perc               | hostname: monasca                         |
|                             | service: monitoring                       |
| cpu.wait_perc               | hostname: monasca                         |
|                             | service: monitoring                       |
| disk.inode_used_perc        | device: /                                 |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| disk.space_used_perc        | device: /                                 |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| http_status                 | url: http://localhost:8081/healthcheck    |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| io.read_kbytes_sec          | device: vda                               |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| io.read_req_sec             | device: vda                               |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| io.write_kbytes_sec         | device: vda                               |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| io.write_req_sec            | device: vda                               |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| load.avg_15_min             | hostname: monasca                         |
|                             | service: monitoring                       |
| load.avg_1_min              | hostname: monasca                         |
|                             | service: monitoring                       |
| load.avg_5_min              | hostname: monasca                         |
|                             | service: monitoring                       |
| mem.free_mb                 | hostname: monasca                         |
|                             | service: monitoring                       |
| mem.swap_free_mb            | hostname: monasca                         |
|                             | service: monitoring                       |
| mem.swap_free_perc          | hostname: monasca                         |
|                             | service: monitoring                       |
| mem.swap_total_mb           | hostname: monasca                         |
|                             | service: monitoring                       |
| mem.swap_used_mb            | hostname: monasca                         |
|                             | service: monitoring                       |
| mem.total_mb                | hostname: monasca                         |
|                             | service: monitoring                       |
| mem.usable_mb               | hostname: monasca                         |
|                             | service: monitoring                       |
| mem.usable_perc             | hostname: monasca                         |
|                             | service: monitoring                       |
| monasca.collection_time_sec | hostname: monasca                         |
|                             | component: monasca-agent                  |
|                             | service: monitoring                       |
| monasca.emit_time_sec       | hostname: monasca                         |
|                             | component: monasca-agent                  |
|                             | service: monitoring                       |
| monasca.thread_count        | hostname: monasca                         |
|                             | component: monasca-agent                  |
|                             | service: monitoring                       |
| net.in_bytes                | device: eth0                              |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| net.in_bytes                | device: lo                                |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| net.in_errors               | device: eth0                              |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| net.in_packets              | device: eth0                              |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| net.out_bytes               | device: eth0                              |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| net.out_bytes               | device: lo                                |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| net.out_errors              | device: eth0                              |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| net.out_packets             | device: eth0                              |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| process.pid_count           | process_name: kafka.Kafka                 |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| process.pid_count           | process_name: mysqld                      |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| process.pid_count           | process_name: org.apache.zookeeper.server |
|                             | hostname: monasca                         |
|                             | service: monitoring                       |
| zookeeper.avg_latency_sec   | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
| zookeeper.connections_count | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
| zookeeper.in_bytes          | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
| zookeeper.max_latency_sec   | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
| zookeeper.min_latency_sec   | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
| zookeeper.node_count        | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
| zookeeper.out_bytes         | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
| zookeeper.outstanding_bytes | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
| zookeeper.zxid_count        | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
| zookeeper.zxid_epoch        | hostname: monasca                         |
|                             | component: zookeeper                      |
|                             | mode: standalone                          |
|                             | service: monitoring                       |
+-----------------------------+-------------------------------------------+

Now let’s create an alarm definition. First we’ll create a webhook notification and then the actual alarm definition.

ubuntu@devstack:~$ monasca notification-create testWebhook WEBHOOK http://testlocation
{
  "address": "http://testlocation",
  "type": "WEBHOOK",
  "id": "b3cac6ec-3976-4984-8acc-28b81a6c7ca2",
  "links": [
    {
      "href": "http://monasca:8080/v2.0/notification-methods/b3cac6ec-3976-4984-8acc-28b81a6c7ca2",
      "rel": "self"
    }
  ],
  "name": "testWebhook"
}
ubuntu@devstack:~$ monasca alarm-definition-create --alarm-actions b3cac6ec-3976-4984-8acc-28b81a6c7ca2 TestAlarmCPU "AVG(cpu.system_perc) > 1"
{
  "alarm_actions": [
    "b3cac6ec-3976-4984-8acc-28b81a6c7ca2"
  ],
  "ok_actions": [],
  "description": "",
  "links": [
    {
      "href": "http://monasca:8080/v2.0/alarm-definitions/a115ff73-b139-4693-8aa2-a63031df723c",
      "rel": "self"
    }
  ],
  "match_by": [],
  "severity": "LOW",
  "actions_enabled": true,
  "undetermined_actions": [],
  "expression": "AVG(cpu.system_perc) > 1",
  "id": "a115ff73-b139-4693-8aa2-a63031df723c",
  "name": "TestAlarmCPU"
}
ubuntu@devstack:~$ monasca alarm-definition-list
+-----------+--------------------------------------+---------------------------+----------+-----------------+
| name      | id                                   | expression                | match_by | actions_enabled |
+-----------+--------------------------------------+---------------------------+----------+-----------------+
| estAlarmCPU | a115ff73-b139-4693-8aa2-a63031df723c | AVG(cpu.system_perc) > 1 |          | True            |
+-----------+--------------------------------------+---------------------------+----------+-----------------+

After a minute or two test to see what the alarm state is.

ubuntu@devstack:~$ monasca alarm-list
+--------------------------------------+--------------------------------------+--------------+-----------------+---------------------+----------+-------+
| id                                   | alarm_definition_id                  | alarm_name   | metric_name     | metric_dimensions   | severity | state |
+--------------------------------------+--------------------------------------+--------------+-----------------+---------------------+----------+-------+
| a744fe79-8330-4525-91fb-3cb1255f7095 | a115ff73-b139-4693-8aa2-a63031df723c | TestAlarmCPU | cpu.system_perc | hostname: monasca   | LOW      | ALARM |
|                                      |                                      |              |                 | service: monitoring |          |       |
+--------------------------------------+--------------------------------------+--------------+-----------------+---------------------+----------+-------+

In this case an alarm was triggered as I made some load on the machine.

This should cover the installation and configuration of Monasca in Openstack (Devstack). If you have any problems, feel free to contact me.

6 thoughts on “How to install and setup Monasca (3/3)

  1. Hi, some help needed, after hard time i was able to install the all component. Now i’m facing problem creating new alarm from horizon ui (404 page not found), here is my api log sample, any change on help ?

    DEBUG [2015-05-07 11:58:33,192] org.eclipse.jetty.io.SelectorManager: Destroyed SelectChannelEndPoint@2539efe5{/135.248.18.185:51000/10.38.231.90:8080,o=false,is=true,os=true,fi=FillInterest@7eb4b67a{false,null},wf=WriteFlusher@a384493{IDLE},it=30000}{HttpConnection@741e1422{FILLING},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of -1}}{io=0,kio=-1,kro=-1}
    DEBUG [2015-05-07 11:58:33,192] org.eclipse.jetty.io.AbstractConnection: onClose HttpConnection@741e1422{FILLING},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of -1}
    DEBUG [2015-05-07 11:58:33,193] org.eclipse.jetty.io.SelectorManager: Selector loop waiting on select
    DEBUG [2015-05-07 11:58:33,197] org.eclipse.jetty.io.AbstractEndPoint: onClose SelectChannelEndPoint@2539efe5{/135.248.18.185:51000/10.38.231.90:8080,o=false,is=true,os=true,fi=FillInterest@7eb4b67a{false,null},wf=WriteFlusher@a384493{IDLE},it=30000}{HttpConnection@741e1422{FILLING},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of -1}}{io=0,kio=-1,kro=-1}
    DEBUG [2015-05-07 11:58:33,200] monasca.api.infrastructure.persistence.influxdb.InfluxV9MetricDefinitionRepo: Metric definition query: show series where _tenant_id=’09305a46ccab41ddaa3f97765cbe151b’ and _region=’RegionOne’ limit 1 offset 0
    DEBUG [2015-05-07 11:58:33,204] monasca.api.infrastructure.persistence.influxdb.InfluxV9RepoReader: Sending query show series where _tenant_id=’09305a46ccab41ddaa3f97765cbe151b’ and _region=’RegionOne’ limit 1 offset 0 to influx database mon at http://10.38.231.90:8086/query
    DEBUG [2015-05-07 11:58:33,215] org.apache.http.client.protocol.RequestAddCookies: CookieSpec selected: default
    DEBUG [2015-05-07 11:58:33,222] org.apache.http.client.protocol.RequestAuthCache: Auth cache not set in the context
    DEBUG [2015-05-07 11:58:33,223] org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Connection request: [route: {}->http://10.38.231.90:8086%5D%5Btotal kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 100]
    DEBUG [2015-05-07 11:58:33,234] org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://10.38.231.90:8086%5D%5Btotal kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 100]
    DEBUG [2015-05-07 11:58:33,235] org.apache.http.impl.execchain.MainClientExec: Opening connection {}->http://10.38.231.90:8086
    DEBUG [2015-05-07 11:58:33,237] org.apache.http.impl.conn.DefaultHttpClientConnectionOperator: Connecting to /10.38.231.90:8086
    DEBUG [2015-05-07 11:58:33,238] org.apache.http.impl.conn.DefaultHttpClientConnectionOperator: Connection established 10.38.231.90:4215410.38.231.90:8086
    DEBUG [2015-05-07 11:58:33,238] org.apache.http.impl.execchain.MainClientExec: Executing request GET /query?q=show+series++where++_tenant_id%3D%2709305a46ccab41ddaa3f97765cbe151b%27++and+_region%3D%27RegionOne%27+++limit+1++offset+0&db=mon HTTP/1.1
    DEBUG [2015-05-07 11:58:33,238] org.apache.http.impl.execchain.MainClientExec: Proxy auth state: UNCHALLENGED
    DEBUG [2015-05-07 11:58:33,239] org.apache.http.headers: http-outgoing-0 >> GET /query?q=show+series++where++_tenant_id%3D%2709305a46ccab41ddaa3f97765cbe151b%27++and+_region%3D%27RegionOne%27+++limit+1++offset+0&db=mon HTTP/1.1
    DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> content-type: application/json
    DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> Authorization: Basic bW9uX2FwaTpyb290
    DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> Accept-Encoding: gzip
    DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> Host: 10.38.231.90:8086
    DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> Connection: Keep-Alive
    DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> User-Agent: Apache-HttpClient/4.4 (Java 1.5 minimum; Java/1.7.0_79)
    DEBUG [2015-05-07 11:58:33,240] org.apache.http.wire: http-outgoing-0 >> “GET /query?q=show+series++where++_tenant_id%3D%2709305a46ccab41ddaa3f97765cbe151b%27++and+_region%3D%27RegionOne%27+++limit+1++offset+0&db=mon HTTP/1.1[\r][\n]”
    DEBUG [2015-05-07 11:58:33,240] org.apache.http.wire: http-outgoing-0 >> “content-type: application/json[\r][\n]”
    DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “Authorization: Basic bW9uX2FwaTpyb290[\r][\n]”
    DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “Accept-Encoding: gzip[\r][\n]”
    DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “Host: 10.38.231.90:8086[\r][\n]”
    DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “Connection: Keep-Alive[\r][\n]”
    DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “User-Agent: Apache-HttpClient/4.4 (Java 1.5 minimum; Java/1.7.0_79)[\r][\n]”
    DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “[\r][\n]”
    DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
    DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "Content-Type: text/plain; charset=utf-8[\r][\n]"
    DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "Date: Thu, 07 May 2015 11:58:33 GMT[\r][\n]"
    DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "Content-Length: 19[\r][\n]"
    DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "[\r][\n]"
    DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "404 page not found[\n]"
    DEBUG [2015-05-07 11:58:33,244] org.apache.http.headers: http-outgoing-0 << HTTP/1.1 404 Not Found
    DEBUG [2015-05-07 11:58:33,244] org.apache.http.headers: http-outgoing-0 << Content-Type: text/plain; charset=utf-8
    DEBUG [2015-05-07 11:58:33,244] org.apache.http.headers: http-outgoing-0 << Date: Thu, 07 May 2015 11:58:33 GMT
    DEBUG [2015-05-07 11:58:33,245] org.apache.http.headers: http-outgoing-0 <http://10.38.231.90:8086%5D can be kept alive indefinitely
    DEBUG [2015-05-07 11:58:33,251] org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://10.38.231.90:8086%5D%5Btotal kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 100]
    ERROR [2015-05-07 11:58:33,251] monasca.api.infrastructure.persistence.influxdb.InfluxV9RepoReader: Failed to query influx database mon at http://10.38.231.90:8086/query: 404
    ERROR [2015-05-07 11:58:33,251] monasca.api.infrastructure.persistence.influxdb.InfluxV9RepoReader: Http response: 404 page not found

      • I also have the same problem after installed all component. Seem like the the GUI to access manasca api component is denied. Anything I have to do to make it work?
        The error when I click the alarm definition button is:
        Error: Could not retrieve alarm definitions
        Thanks for any comments.

  2. Hi ,I have installed monasca and its components.Able to see system metrics and process metrics in grafana dashboard.
    Since I have installed monasca-agent on one machine and other monasca services on another, while running the monasca-setup , it dint pick other services like Mysql, Kafka ,monasca components like Thresh ,Storm , Notification ,Influx.
    I was able to configure zookeeper though by creating a config file (.yaml in etc/monasca/agent/conf.d ) Tried a similar way for Kafka and Mysql using example config files created in /usr/local/share/monasca/agent/conf.d. But still collector is not collecting the metrics for mysql ,kafka . Also unable to find sample yaml config file for Mon plugin (For measuring mon-persister, mon-api and mon-thresh).

    Let me know if any other detail is needed.
    Thanks for your help

    • mysql.yaml

      init_config:

      instances:
      – server: IP
      user: root
      pass: password
      port: 3306

      Kafka_coonsumer.yaml

      init_config:

      instances:

      – kafka_connect_str: IP:9092
      zk_connect_str: IP:2181

  3. Hi,

    The webhook which I am trying to create needs authentication to call it ( needs auth header and other headers). How can i specify them? Also, is there a way to run a script which can do calling APIs instead directly calling webhook?

Leave a Reply

Your email address will not be published. Required fields are marked *