In this final part of the tutorial we will verify that all things are working properly.
Because of the large number of moving components the norm is that things will go wrong. Code changes daily both in Monasca and in Openstack and quite often things need further adjusting.
As a general rule, in case a component is misbehaving, change logging from INFO to DEBUG in /etc/monasca and restart the component. The look at the logs in /var/log/monasca and see what is wrong. For Thresh / Storm, logs can be found in /opt/storm/current/logs
In order to test if monasca-ui is properly configured log into Horizon with
username: monasca password: password
And the monitoring panel should be visible. In case it isn’t, monasca-ui might not be getting what it expects from Keystone.
In order to test monasca-api, first thing you need to do is get a valid keystone toke (let’s say for admin). Then, on a machine which has the monasca client installed run:
export OS_USERNAME=monasca export OS_PASSWORD=password export OS_AUTH_URL=http://devstack:5000/v3 monasca alarm-definition-list
That should return nothing as no definitions are present. Next look at some of the metrics which are being collected.
ubuntu@devstack:~$ monasca metric-list +-----------------------------+-------------------------------------------+ | name | dimensions | +-----------------------------+-------------------------------------------+ | cpu.idle_perc | hostname: monasca | | | service: monitoring | | cpu.stolen_perc | hostname: monasca | | | service: monitoring | | cpu.system_perc | hostname: monasca | | | service: monitoring | | cpu.user_perc | hostname: monasca | | | service: monitoring | | cpu.wait_perc | hostname: monasca | | | service: monitoring | | disk.inode_used_perc | device: / | | | hostname: monasca | | | service: monitoring | | disk.space_used_perc | device: / | | | hostname: monasca | | | service: monitoring | | http_status | url: http://localhost:8081/healthcheck | | | hostname: monasca | | | service: monitoring | | io.read_kbytes_sec | device: vda | | | hostname: monasca | | | service: monitoring | | io.read_req_sec | device: vda | | | hostname: monasca | | | service: monitoring | | io.write_kbytes_sec | device: vda | | | hostname: monasca | | | service: monitoring | | io.write_req_sec | device: vda | | | hostname: monasca | | | service: monitoring | | load.avg_15_min | hostname: monasca | | | service: monitoring | | load.avg_1_min | hostname: monasca | | | service: monitoring | | load.avg_5_min | hostname: monasca | | | service: monitoring | | mem.free_mb | hostname: monasca | | | service: monitoring | | mem.swap_free_mb | hostname: monasca | | | service: monitoring | | mem.swap_free_perc | hostname: monasca | | | service: monitoring | | mem.swap_total_mb | hostname: monasca | | | service: monitoring | | mem.swap_used_mb | hostname: monasca | | | service: monitoring | | mem.total_mb | hostname: monasca | | | service: monitoring | | mem.usable_mb | hostname: monasca | | | service: monitoring | | mem.usable_perc | hostname: monasca | | | service: monitoring | | monasca.collection_time_sec | hostname: monasca | | | component: monasca-agent | | | service: monitoring | | monasca.emit_time_sec | hostname: monasca | | | component: monasca-agent | | | service: monitoring | | monasca.thread_count | hostname: monasca | | | component: monasca-agent | | | service: monitoring | | net.in_bytes | device: eth0 | | | hostname: monasca | | | service: monitoring | | net.in_bytes | device: lo | | | hostname: monasca | | | service: monitoring | | net.in_errors | device: eth0 | | | hostname: monasca | | | service: monitoring | | net.in_packets | device: eth0 | | | hostname: monasca | | | service: monitoring | | net.out_bytes | device: eth0 | | | hostname: monasca | | | service: monitoring | | net.out_bytes | device: lo | | | hostname: monasca | | | service: monitoring | | net.out_errors | device: eth0 | | | hostname: monasca | | | service: monitoring | | net.out_packets | device: eth0 | | | hostname: monasca | | | service: monitoring | | process.pid_count | process_name: kafka.Kafka | | | hostname: monasca | | | service: monitoring | | process.pid_count | process_name: mysqld | | | hostname: monasca | | | service: monitoring | | process.pid_count | process_name: org.apache.zookeeper.server | | | hostname: monasca | | | service: monitoring | | zookeeper.avg_latency_sec | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | | zookeeper.connections_count | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | | zookeeper.in_bytes | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | | zookeeper.max_latency_sec | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | | zookeeper.min_latency_sec | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | | zookeeper.node_count | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | | zookeeper.out_bytes | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | | zookeeper.outstanding_bytes | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | | zookeeper.zxid_count | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | | zookeeper.zxid_epoch | hostname: monasca | | | component: zookeeper | | | mode: standalone | | | service: monitoring | +-----------------------------+-------------------------------------------+
Now let’s create an alarm definition. First we’ll create a webhook notification and then the actual alarm definition.
ubuntu@devstack:~$ monasca notification-create testWebhook WEBHOOK http://testlocation
{
"address": "http://testlocation",
"type": "WEBHOOK",
"id": "b3cac6ec-3976-4984-8acc-28b81a6c7ca2",
"links": [
{
"href": "http://monasca:8080/v2.0/notification-methods/b3cac6ec-3976-4984-8acc-28b81a6c7ca2",
"rel": "self"
}
],
"name": "testWebhook"
}
ubuntu@devstack:~$ monasca alarm-definition-create --alarm-actions b3cac6ec-3976-4984-8acc-28b81a6c7ca2 TestAlarmCPU "AVG(cpu.system_perc) > 1"
{
"alarm_actions": [
"b3cac6ec-3976-4984-8acc-28b81a6c7ca2"
],
"ok_actions": [],
"description": "",
"links": [
{
"href": "http://monasca:8080/v2.0/alarm-definitions/a115ff73-b139-4693-8aa2-a63031df723c",
"rel": "self"
}
],
"match_by": [],
"severity": "LOW",
"actions_enabled": true,
"undetermined_actions": [],
"expression": "AVG(cpu.system_perc) > 1",
"id": "a115ff73-b139-4693-8aa2-a63031df723c",
"name": "TestAlarmCPU"
}
ubuntu@devstack:~$ monasca alarm-definition-list
+-----------+--------------------------------------+---------------------------+----------+-----------------+
| name | id | expression | match_by | actions_enabled |
+-----------+--------------------------------------+---------------------------+----------+-----------------+
| estAlarmCPU | a115ff73-b139-4693-8aa2-a63031df723c | AVG(cpu.system_perc) > 1 | | True |
+-----------+--------------------------------------+---------------------------+----------+-----------------+
After a minute or two test to see what the alarm state is.
ubuntu@devstack:~$ monasca alarm-list +--------------------------------------+--------------------------------------+--------------+-----------------+---------------------+----------+-------+ | id | alarm_definition_id | alarm_name | metric_name | metric_dimensions | severity | state | +--------------------------------------+--------------------------------------+--------------+-----------------+---------------------+----------+-------+ | a744fe79-8330-4525-91fb-3cb1255f7095 | a115ff73-b139-4693-8aa2-a63031df723c | TestAlarmCPU | cpu.system_perc | hostname: monasca | LOW | ALARM | | | | | | service: monitoring | | | +--------------------------------------+--------------------------------------+--------------+-----------------+---------------------+----------+-------+
In this case an alarm was triggered as I made some load on the machine.
This should cover the installation and configuration of Monasca in Openstack (Devstack). If you have any problems, feel free to contact me.
Hi, some help needed, after hard time i was able to install the all component. Now i’m facing problem creating new alarm from horizon ui (404 page not found), here is my api log sample, any change on help ?
DEBUG [2015-05-07 11:58:33,192] org.eclipse.jetty.io.SelectorManager: Destroyed SelectChannelEndPoint@2539efe5{/135.248.18.185:51000/10.38.231.90:8080,o=false,is=true,os=true,fi=FillInterest@7eb4b67a{false,null},wf=WriteFlusher@a384493{IDLE},it=30000}{HttpConnection@741e1422{FILLING},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of -1}}{io=0,kio=-1,kro=-1}
DEBUG [2015-05-07 11:58:33,192] org.eclipse.jetty.io.AbstractConnection: onClose HttpConnection@741e1422{FILLING},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of -1}
DEBUG [2015-05-07 11:58:33,193] org.eclipse.jetty.io.SelectorManager: Selector loop waiting on select
DEBUG [2015-05-07 11:58:33,197] org.eclipse.jetty.io.AbstractEndPoint: onClose SelectChannelEndPoint@2539efe5{/135.248.18.185:51000/10.38.231.90:8080,o=false,is=true,os=true,fi=FillInterest@7eb4b67a{false,null},wf=WriteFlusher@a384493{IDLE},it=30000}{HttpConnection@741e1422{FILLING},g=HttpGenerator{s=START},p=HttpParser{s=START,0 of -1}}{io=0,kio=-1,kro=-1}
DEBUG [2015-05-07 11:58:33,200] monasca.api.infrastructure.persistence.influxdb.InfluxV9MetricDefinitionRepo: Metric definition query: show series where _tenant_id=’09305a46ccab41ddaa3f97765cbe151b’ and _region=’RegionOne’ limit 1 offset 0
DEBUG [2015-05-07 11:58:33,204] monasca.api.infrastructure.persistence.influxdb.InfluxV9RepoReader: Sending query show series where _tenant_id=’09305a46ccab41ddaa3f97765cbe151b’ and _region=’RegionOne’ limit 1 offset 0 to influx database mon at http://10.38.231.90:8086/query
DEBUG [2015-05-07 11:58:33,215] org.apache.http.client.protocol.RequestAddCookies: CookieSpec selected: default
DEBUG [2015-05-07 11:58:33,222] org.apache.http.client.protocol.RequestAuthCache: Auth cache not set in the context
DEBUG [2015-05-07 11:58:33,223] org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Connection request: [route: {}->http://10.38.231.90:8086%5D%5Btotal kept alive: 0; route allocated: 0 of 2; total allocated: 0 of 100]
DEBUG [2015-05-07 11:58:33,234] org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Connection leased: [id: 0][route: {}->http://10.38.231.90:8086%5D%5Btotal kept alive: 0; route allocated: 1 of 2; total allocated: 1 of 100]
DEBUG [2015-05-07 11:58:33,235] org.apache.http.impl.execchain.MainClientExec: Opening connection {}->http://10.38.231.90:8086
DEBUG [2015-05-07 11:58:33,237] org.apache.http.impl.conn.DefaultHttpClientConnectionOperator: Connecting to /10.38.231.90:8086
DEBUG [2015-05-07 11:58:33,238] org.apache.http.impl.conn.DefaultHttpClientConnectionOperator: Connection established 10.38.231.90:4215410.38.231.90:8086
DEBUG [2015-05-07 11:58:33,238] org.apache.http.impl.execchain.MainClientExec: Executing request GET /query?q=show+series++where++_tenant_id%3D%2709305a46ccab41ddaa3f97765cbe151b%27++and+_region%3D%27RegionOne%27+++limit+1++offset+0&db=mon HTTP/1.1
DEBUG [2015-05-07 11:58:33,238] org.apache.http.impl.execchain.MainClientExec: Proxy auth state: UNCHALLENGED
DEBUG [2015-05-07 11:58:33,239] org.apache.http.headers: http-outgoing-0 >> GET /query?q=show+series++where++_tenant_id%3D%2709305a46ccab41ddaa3f97765cbe151b%27++and+_region%3D%27RegionOne%27+++limit+1++offset+0&db=mon HTTP/1.1
DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> content-type: application/json
DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> Authorization: Basic bW9uX2FwaTpyb290
DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> Accept-Encoding: gzip
DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> Host: 10.38.231.90:8086
DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> Connection: Keep-Alive
DEBUG [2015-05-07 11:58:33,240] org.apache.http.headers: http-outgoing-0 >> User-Agent: Apache-HttpClient/4.4 (Java 1.5 minimum; Java/1.7.0_79)
DEBUG [2015-05-07 11:58:33,240] org.apache.http.wire: http-outgoing-0 >> “GET /query?q=show+series++where++_tenant_id%3D%2709305a46ccab41ddaa3f97765cbe151b%27++and+_region%3D%27RegionOne%27+++limit+1++offset+0&db=mon HTTP/1.1[\r][\n]”
DEBUG [2015-05-07 11:58:33,240] org.apache.http.wire: http-outgoing-0 >> “content-type: application/json[\r][\n]”
DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “Authorization: Basic bW9uX2FwaTpyb290[\r][\n]”
DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “Accept-Encoding: gzip[\r][\n]”
DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “Host: 10.38.231.90:8086[\r][\n]”
DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “Connection: Keep-Alive[\r][\n]”
DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “User-Agent: Apache-HttpClient/4.4 (Java 1.5 minimum; Java/1.7.0_79)[\r][\n]”
DEBUG [2015-05-07 11:58:33,241] org.apache.http.wire: http-outgoing-0 >> “[\r][\n]”
DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "HTTP/1.1 404 Not Found[\r][\n]"
DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "Content-Type: text/plain; charset=utf-8[\r][\n]"
DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "Date: Thu, 07 May 2015 11:58:33 GMT[\r][\n]"
DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "Content-Length: 19[\r][\n]"
DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "[\r][\n]"
DEBUG [2015-05-07 11:58:33,242] org.apache.http.wire: http-outgoing-0 << "404 page not found[\n]"
DEBUG [2015-05-07 11:58:33,244] org.apache.http.headers: http-outgoing-0 << HTTP/1.1 404 Not Found
DEBUG [2015-05-07 11:58:33,244] org.apache.http.headers: http-outgoing-0 << Content-Type: text/plain; charset=utf-8
DEBUG [2015-05-07 11:58:33,244] org.apache.http.headers: http-outgoing-0 << Date: Thu, 07 May 2015 11:58:33 GMT
DEBUG [2015-05-07 11:58:33,245] org.apache.http.headers: http-outgoing-0 <http://10.38.231.90:8086%5D can be kept alive indefinitely
DEBUG [2015-05-07 11:58:33,251] org.apache.http.impl.conn.PoolingHttpClientConnectionManager: Connection released: [id: 0][route: {}->http://10.38.231.90:8086%5D%5Btotal kept alive: 1; route allocated: 1 of 2; total allocated: 1 of 100]
ERROR [2015-05-07 11:58:33,251] monasca.api.infrastructure.persistence.influxdb.InfluxV9RepoReader: Failed to query influx database mon at http://10.38.231.90:8086/query: 404
ERROR [2015-05-07 11:58:33,251] monasca.api.infrastructure.persistence.influxdb.InfluxV9RepoReader: Http response: 404 page not found
Hi Alon,
I will contact you personally to solve this problem.
Victor
I also have the same problem after installed all component. Seem like the the GUI to access manasca api component is denied. Anything I have to do to make it work?
The error when I click the alarm definition button is:
Error: Could not retrieve alarm definitions
Thanks for any comments.
Hi ,I have installed monasca and its components.Able to see system metrics and process metrics in grafana dashboard.
Since I have installed monasca-agent on one machine and other monasca services on another, while running the monasca-setup , it dint pick other services like Mysql, Kafka ,monasca components like Thresh ,Storm , Notification ,Influx.
I was able to configure zookeeper though by creating a config file (.yaml in etc/monasca/agent/conf.d ) Tried a similar way for Kafka and Mysql using example config files created in /usr/local/share/monasca/agent/conf.d. But still collector is not collecting the metrics for mysql ,kafka . Also unable to find sample yaml config file for Mon plugin (For measuring mon-persister, mon-api and mon-thresh).
Let me know if any other detail is needed.
Thanks for your help
mysql.yaml
init_config:
instances:
– server: IP
user: root
pass: password
port: 3306
Kafka_coonsumer.yaml
init_config:
instances:
– kafka_connect_str: IP:9092
zk_connect_str: IP:2181
Hi,
The webhook which I am trying to create needs authentication to call it ( needs auth header and other headers). How can i specify them? Also, is there a way to run a script which can do calling APIs instead directly calling webhook?