{"id":6675,"date":"2014-11-13T19:26:53","date_gmt":"2014-11-13T17:26:53","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=6675"},"modified":"2015-02-05T10:52:42","modified_gmt":"2015-02-05T08:52:42","slug":"monasca-for-cloud-monitoring-initial-impressions","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/","title":{"rendered":"Monasca for Cloud Monitoring: Initial impressions"},"content":{"rendered":"<p>One of the focuses of the Cloud Incident Management research initiative are Monitoring as a Service solutions as they provide the building blocks for incident detection and resolution. As such, part of the work carried throughout the initiative was on identifying good, maintainable, monitoring solutions which can be easily adapted and integrated into a greater incident management architecture. The current blog post tries to cover Monasca showing the good, the bad and the ugly.<\/p>\n<p>Monasca is a Monitoring as a Service solution which comes from HP and Rackspace: it focuses on providing a complete monitoring solution for Openstack. Monasca is an open source solution designed to be highly scalable, performant and fault-tolerant for a multi-tenant environment. It features a RESTful API though which one can interact with the system in order to query it or send metrics for processing.<\/p>\n<p>The solution monitors both the Openstack Infrastructure as well as the VMs which run on it. Further, it can be easily integrated with <a href=\"https:\/\/github.com\/rackerlabs\/stacktach\" target=\"_blank\">Rackspace&#8217;s Stacktach<\/a> which forwards all Openstack events coming from its different components to be processed by Monasca. Additionally, the primary authentication mechanism it uses as well as service catalog is Keystone.<\/p>\n<p><!--more--><\/p>\n<p>The <a href=\"https:\/\/wiki.openstack.org\/wiki\/Monasca\">general architecture<\/a> is designed from the ground up to provide stream processing of metrics and events which are sent via <a href=\"http:\/\/kafka.apache.org\/\" target=\"_blank\">Kafka<\/a> which acts as a general cloud bus through the publish-subscribe pattern.<\/p>\n<p>The following are the core components of Monasca:<\/p>\n<ul>\n<li><a href=\"https:\/\/github.com\/stackforge\/monasca-api\">Monasca API<\/a> &#8212; acts as a gateway to the whole system\u00a0and can be interacted with via\u00a0the RESTful API. It supports\u00a0the management of alarms definitions, notification methods and management of metrics (storage, querying).<\/li>\n<li><a href=\"https:\/\/github.com\/stackforge\/python-monascaclient\">Monasca Client<\/a> &#8212; is a python-based command line client and library used which can be used for easy communication with Monasca<\/li>\n<li><a href=\"https:\/\/github.com\/stackforge\/monasca-agent\">Monasca Agent<\/a> &#8212; retrieves metrics from the host on which it runs. It comes with a wide range of plugins and even supports the usage of Nagios plugins. It is made up of 3 subcomponents:\n<ul>\n<li>Collector &#8212; collects the metrics,<\/li>\n<li>Monstatsd &#8212; a <a href=\"https:\/\/github.com\/etsy\/statsd\/\">StatsD<\/a> compatible solution for metrics coming from other sources,<\/li>\n<li>Forwarder &#8212; which takes the metrics from the Collector and Monstatsd, performs normalization if necessary, and forwards everything to the Monasca API.<\/li>\n<\/ul>\n<\/li>\n<li><a href=\"https:\/\/github.com\/stackforge\/monasca-persister\">Monasca Persister<\/a> &#8212; handles everything which has to do with long term storage of metrics and alarms in distributed databases such as\u00a0<a href=\"http:\/\/influxdb.com\/\">InfluxDB<\/a> or analytics platforms such as\u00a0<a href=\"http:\/\/www.vertica.com\/\">Vertica<\/a>.<\/li>\n<li><a href=\"https:\/\/github.com\/stackforge\/monasca-thresh\">Monasca Thresholding Engine<\/a> &#8212; performs analysis on the incoming metrics and evaluates alarm definitions based on them. Any triggered alarms are published to Kafka so that the notification process can begin. This engine is based on <a href=\"https:\/\/storm.apache.org\/\" target=\"_blank\">Apache Storm<\/a> as it is excellent for stream processing of data.<\/li>\n<li><a href=\"https:\/\/github.com\/stackforge\/monasca-notification\">Monasca Notification Engine<\/a> &#8212; performs notification based on triggered alarms. Currently it supports only email notification.<\/li>\n<li><a href=\"https:\/\/github.com\/stackforge\/monasca-ui\">Monasca UI<\/a> &#8212; is an user interface which is built into Openstack and supports\u00a0browser based interaction with the\u00a0Monasca<\/li>\n<\/ul>\n<p>An interesting component which is under active development is Anomaly Detection to support unsupervised monitoring and detection of problems without needing\u00a0to define rules. It is focused\u00a0on using two specific algorithms: <a href=\"http:\/\/numenta.com\/\">Numenta Platform for Intelligent Computing (NuPIC)<\/a>, which is neural computing type algorithm, and a Kolmogorov\u2013Smirnov which is a two sample nonparametric fitness test. It is receiving a lot of interest right now.<\/p>\n<p>As a whole, Monasca is made up of many moving parts, some written in Java, others in Python. However, as \u00a0further integration with Openstack is a key priority &#8211; indeed it will soon be adopted as an official Openstack project &#8211;\u00a0some of the components are being rewritten in python. There are, of course, parts like the Thresholding Engine that can not be rewritten because of their dependency on different libraries &#8212; in this particular case, dependency on Apache Storm.<\/p>\n<p>A crucial part of the architecture is the way metrics, alarm definitions and alarms can be managed (e.g. created, updated, deleted). This functionality runs throughout the components each handling bits and pieces and is built in the RESTful API.<\/p>\n<p>Metrics in Monasca allow great flexibility in their definition and, consequently, in their processing. As such, a metric has a name, dimensions, value and timestamp. The dimensions enable the metric to be easier classified, for example instance_id:4, service:client_portal are valid dimensions for a metric named cpu.average_5_mins.<\/p>\n<p>The same can be said about alarms definitions. Alarm definitions allow you to specify an expression which will be evaluated based on the metrics received and as a consequence of that an alarm will be created. An example alarm definition expression could be:<\/p>\n<blockquote>\n<pre>max(cpu.load_avg_5_min) &gt; 75<\/pre>\n<\/blockquote>\n<p>Alarm definitions determine the creation of alarms only when their expression can be processed against valid metrics. Alarms start with an UNDEFINED state which is changed to OK as long as the expression is not triggered. Once it is, the state changes to ALARM and the Notification Engine will start notifying people.<\/p>\n<p>While Monasca has\u00a0many nice features, it is a complex piece of software and suffers from some of the problems of large software projects which are not yet mature. As it is still under active development, unexpected behaviour is to be expected!<\/p>\n<p>The following are some of the things I stumbled upon while trying to work with it:<\/p>\n<ul>\n<li>Lack of documentation &#8212; The project lacks a lot of documentation about the API, the metrics, alarms, and generally\u00a0how things work. Recently, more information has been added, but there are still many holes in the documentation.<\/li>\n<li>Difficult installation and testing process &#8212; while there is a Vagrant installation available, when you need to deploy it on a dev environment or even production, there is no easy way to do it and no documentation about how to do it is available.<\/li>\n<li>Code problems &#8212; while performing the installation i ran into many problems like hard coded links, lack of naming conventions, quirky support for authentication (needs the Openstack Keystone service key) etc.<\/li>\n<li>Minimal support for logs &#8212; right now, only log statistics are being processed and sent as metrics and most of the time more is needed.<\/li>\n<\/ul>\n<p>For those not involved with Openstack who want to make use of Monasca, the only real requirement is Keystone which can be separately installed.<\/p>\n<p>Overall, Monasca is a great technology but it is still under heavy development. People who\u00a0like to play with new technology can go ahead and try it. A list of the complete documentation, wiki and repositories for Monasca can be found <a href=\"https:\/\/launchpad.net\/monasca\" target=\"_blank\">here<\/a>.<\/p>\n<p>The next Monasca blog post will cover installation of Monasca in a typical environment.<\/p>\n<div class=\"pt-sm\">Schlagw\u00f6rter: <a href=\"https:\/\/blog.zhaw.ch\/icclab\/tag\/cloud-incident-management-2\/\">cloud incident management<\/a>, <a href=\"https:\/\/blog.zhaw.ch\/icclab\/tag\/monasca\/\">monasca<\/a>, <a href=\"https:\/\/blog.zhaw.ch\/icclab\/tag\/monitoring-as-a-service\/\">monitoring as a service<\/a><br><\/div>","protected":false},"excerpt":{"rendered":"<p>One of the focuses of the Cloud Incident Management research initiative are Monitoring as a Service solutions as they provide the building blocks for incident detection and resolution. As such, part of the work carried throughout the initiative was on identifying good, maintainable, monitoring solutions which can be easily adapted and integrated into a greater [&hellip;]<\/p>\n","protected":false},"author":142,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[1,5],"tags":[509,510,511],"features":[],"class_list":["post-6675","post","type-post","status-publish","format-standard","hentry","category-allgemein","category-articles","tag-cloud-incident-management-2","tag-monasca","tag-monitoring-as-a-service"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Monasca for Cloud Monitoring: Initial impressions - Service Engineering (ICCLab &amp; SPLab)<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Monasca for Cloud Monitoring: Initial impressions\" \/>\n<meta property=\"og:description\" content=\"One of the focuses of the Cloud Incident Management research initiative are Monitoring as a Service solutions as they provide the building blocks for incident detection and resolution. As such, part of the work carried throughout the initiative was on identifying good, maintainable, monitoring solutions which can be easily adapted and integrated into a greater [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/\" \/>\n<meta property=\"og:site_name\" content=\"Service Engineering (ICCLab &amp; SPLab)\" \/>\n<meta property=\"article:published_time\" content=\"2014-11-13T17:26:53+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2015-02-05T08:52:42+00:00\" \/>\n<meta name=\"author\" content=\"mune\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"mune\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"5 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/\"},\"author\":{\"name\":\"mune\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/ad6f9dc95a64a3da5cfcbe9b76581014\"},\"headline\":\"Monasca for Cloud Monitoring: Initial impressions\",\"datePublished\":\"2014-11-13T17:26:53+00:00\",\"dateModified\":\"2015-02-05T08:52:42+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/\"},\"wordCount\":1088,\"commentCount\":0,\"keywords\":[\"cloud incident management\",\"monasca\",\"monitoring as a service\"],\"articleSection\":[\"*.*\",\"Articles\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/\",\"name\":\"Monasca for Cloud Monitoring: Initial impressions - Service Engineering (ICCLab &amp; SPLab)\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\"},\"datePublished\":\"2014-11-13T17:26:53+00:00\",\"dateModified\":\"2015-02-05T08:52:42+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/ad6f9dc95a64a3da5cfcbe9b76581014\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/icclab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Monasca for Cloud Monitoring: Initial impressions\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/\",\"name\":\"Service Engineering (ICCLab &amp; SPLab)\",\"description\":\"A Blog of the ZHAW Zurich University of Applied Sciences\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/ad6f9dc95a64a3da5cfcbe9b76581014\",\"name\":\"mune\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/7b886759523f8a854ac234c59f59f32dee27ca251311c457235251e8aa94bcaf?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/7b886759523f8a854ac234c59f59f32dee27ca251311c457235251e8aa94bcaf?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/7b886759523f8a854ac234c59f59f32dee27ca251311c457235251e8aa94bcaf?s=96&d=mm&r=g\",\"caption\":\"mune\"},\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/author\/mune\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Monasca for Cloud Monitoring: Initial impressions - Service Engineering (ICCLab &amp; SPLab)","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/","og_locale":"en_US","og_type":"article","og_title":"Monasca for Cloud Monitoring: Initial impressions","og_description":"One of the focuses of the Cloud Incident Management research initiative are Monitoring as a Service solutions as they provide the building blocks for incident detection and resolution. As such, part of the work carried throughout the initiative was on identifying good, maintainable, monitoring solutions which can be easily adapted and integrated into a greater [&hellip;]","og_url":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/","og_site_name":"Service Engineering (ICCLab &amp; SPLab)","article_published_time":"2014-11-13T17:26:53+00:00","article_modified_time":"2015-02-05T08:52:42+00:00","author":"mune","twitter_card":"summary_large_image","twitter_misc":{"Written by":"mune","Est. reading time":"5 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/"},"author":{"name":"mune","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/ad6f9dc95a64a3da5cfcbe9b76581014"},"headline":"Monasca for Cloud Monitoring: Initial impressions","datePublished":"2014-11-13T17:26:53+00:00","dateModified":"2015-02-05T08:52:42+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/"},"wordCount":1088,"commentCount":0,"keywords":["cloud incident management","monasca","monitoring as a service"],"articleSection":["*.*","Articles"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/","url":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/","name":"Monasca for Cloud Monitoring: Initial impressions - Service Engineering (ICCLab &amp; SPLab)","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#website"},"datePublished":"2014-11-13T17:26:53+00:00","dateModified":"2015-02-05T08:52:42+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/ad6f9dc95a64a3da5cfcbe9b76581014"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/icclab\/monasca-for-cloud-monitoring-initial-impressions\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/icclab\/"},{"@type":"ListItem","position":2,"name":"Monasca for Cloud Monitoring: Initial impressions"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/icclab\/#website","url":"https:\/\/blog.zhaw.ch\/icclab\/","name":"Service Engineering (ICCLab &amp; SPLab)","description":"A Blog of the ZHAW Zurich University of Applied Sciences","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/ad6f9dc95a64a3da5cfcbe9b76581014","name":"mune","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/7b886759523f8a854ac234c59f59f32dee27ca251311c457235251e8aa94bcaf?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/7b886759523f8a854ac234c59f59f32dee27ca251311c457235251e8aa94bcaf?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/7b886759523f8a854ac234c59f59f32dee27ca251311c457235251e8aa94bcaf?s=96&d=mm&r=g","caption":"mune"},"url":"https:\/\/blog.zhaw.ch\/icclab\/author\/mune\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/6675","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/users\/142"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/comments?post=6675"}],"version-history":[{"count":19,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/6675\/revisions"}],"predecessor-version":[{"id":7511,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/6675\/revisions\/7511"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/media?parent=6675"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/categories?post=6675"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/tags?post=6675"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/features?post=6675"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}