{"id":9648,"date":"2016-01-26T14:37:23","date_gmt":"2016-01-26T12:37:23","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=9648"},"modified":"2016-01-26T17:12:43","modified_gmt":"2016-01-26T15:12:43","slug":"a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/","title":{"rendered":"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System"},"content":{"rendered":"

In one of our projects, FICORE the continuation of FIWARE<\/a>, we need to offer an Openstack-based service. One aspect of service operations is to understand the performance of the system and one particular aspect of this is to understand how long basic operations take; it is interesting to see how this evolves over time as, for example, a system may get more and more loaded. To address this, we first looked at using an approach based on log files but it was not workable as the information regarding an operation is spread across multiple hosts and services. An alternative approach is to use the Openstack notification system where a lot of key events occurring within the system are published – this is a single point for all the information we need. We then used Stacktach to consume, filter and store this data and built a web application on top of it. In this blog post we give a brief overview of the Openstack notification system, the Stacktach filtering tool and the basic web tool we developed.
\n<\/p>\n

OpenStack Notification System<\/strong><\/p>\n

Notifications in OpenStack are primarily fine-grained JSON formatted messages sent to the message queue – usually rabbitmq – containing data relating to a certain operations. The Openstack notification system is generally a quite chatty system which offers a rich data source of structured data which can be used to understand many aspects of Openstack operations. An example of the content of a notification<\/a> is shown below. Each notification has an event_type, usually describing an operation performed – here is a list<\/a> of event_types relating to nova; similar lists exist for other Openstack services.<\/p>\n

{\"event_type\": \"compute.instance.resize.confirm.start\",\r\n      \"timestamp\": \"2012-03-12 17:01:29.899834\",\r\n      \"message_id\": \"1234653e-ce46-4a82-979f-a9286cac5258\",\r\n     \"priority\": \"INFO\",\r\n      \"publisher_id\": \"compute.compute-1\",\r\n      \"payload\": {\"state_description\": \"\",\r\n                           \"display_name\": \"testserver\",\r\n                       \"memory_mb\": 512,\r\n                       \"disk_gb\": 20,\r\n                        \"tenant_id\": \"12345\",\r\n                        \"created_at\": \"2012-03-12 16:55:17\",\r\n                       \"instance_type_id\": 2,\r\n                        \"instance_id\": \"abcbd165-fd41-4fd7-96ac-d70639a042c1\",\r\n                        \"instance_type\": \"512MB instance\",\r\n                        \"state\": \"active\",\r\n                        \"user_id\": \"67890\",\r\n                        \"launched_at\": \"2012-03-12 16:57:29\",\r\n                        \"image_ref_url\": \"http:\/\/127.0.0.1:9292\/images\/a213faf83as0t\"}}\r\n<\/pre>\n
To enable notifications for a certain service add the following lines to its configuration file.<\/p>\n
notification_driver=messaging\r\nnotification_topics=notifications\r\n# In nova.conf\r\nnotify_on_state_change=vm_and_task_state\r\n<\/pre>\n
After applying the changes above (and restarting the service, of course) a new queue will be created in the host if necessary with the name given in notification_topics. Once the services are set up in this way, they will be producing events on the message bus and we can focus on how to work with these notifications using Stacktach to do something useful with them.<\/p>\n
Stacktach<\/strong><\/p>\n
Stacktach comprises of several services focused on consuming, filtering, processing and storing messages for the purposes of analysis or notification: in general, it can be used for any application, but it is really designed for Openstack. More information on Stacktach is available on the official website<\/a> (we really recommend reading the very helpful documentation<\/a>).<\/p>\n
Setting up Stacktach is quite straightforward; here we use the stacktach-sandbox<\/a> within a virtual machine but it can be installed anywhere; importantly, it needs to be installed somewhere that the message queue service of your controller node is accessible. From a vanilla Ubuntu Server VM, it is necessary to install mysql and rabbitmq as follows:<\/p>\n
sudo apt-get install -y python-dev ipython mysql-server mysql-client libmysqlclient-dev git vim rabbitmq-server python-pip librabbitmq1\r\n<\/pre>\n
To install and run Stacktach, the following commands can be run:<\/p>\n
    git clone https:\/\/github.com\/openstack\/stacktach-sandbox.git\r\n    cd stacktach-sandbox\r\n    .\/build.sh\r\n<\/pre>\n
This will create a set of screens<\/a> which contain all of the Stacktach services.<\/p>\n
It is not necessary to have a full Openstack installation to run basic tests on Stacktach; it comes with a notification generator (notigen) which can be used to test the Stacktach deployment and this runs by default when build.sh is executed. Hence, it is possible to determine if the installation has been performed properly by checking for events directly in the database or using the klugman tool in one of the screens:<\/p>\n
    klugman http:\/\/127.0.0.1:8000 streams\r\n<\/pre>\n
Configuring Stacktach to consume Openstack notifications<\/strong><\/p>\n
Before configuring Stacktach, it can be helpful to verify that Openstack is indeed producing notifications: this script<\/a> prints out every message consumed for a given queue. It is useful for debugging Stacktach and helping to understand the workflow of a notification.<\/p>\n
Configuring Stacktach to consume Openstack notifications is fairly straightforward and few modifications need to be done. More specifically, rabbit_broker section in yagi.conf.common must point to the message queue service of your controller node and the database url in winchester.yaml; any other modification will depend on specific configuration of your Openstack deployment, e.g. the notification queue name. For more information relating to Stacktach configuration please follow the documentation<\/a> but here are some caveats to be aware of.<\/p>\n
As the basic deployment of Stacktach consumes notigen messages, it is necessary to modify the configuration to disable consumption of notigen messages. This is done by removing or commenting the following lines from screenrc.winchester:<\/p>\n
    # screen -t gen bash\r\n    # stuff \"cd git\/stacktach-notigen\/bin; python pump_from_stv2.py\\r\"\r\n    # stuff \"cd git\/stacktach-notigen\/bin; python event_pump.py ..\/templates 2 0\\r\"\r\n<\/pre>\n
It is also worth noting that build.sh regenerates yagi.conf from other files every time it is run. This script takes the configuration from yagi.conf.common, yagi.conf.winchester and winchester.yaml within stacktach-sandbox directory.<\/p>\n
Filtering notifications <\/strong><\/p>\n
Stacktach uses a set of rules (triggers) to filter notifications you are interested in and create streams out of them. Streams can be quite arbitrary sequences of events which typically have some initial event and a terminal event. Streams are specific to one or more event types, meaning that a given stream comprises of events which match the given event types. Streams are subdivided using the \u2018distinguished_by\u2019 parameter: this enables us to differentiate between events relating to, for example, specific VMs or hosts. A basic example is shown below which matches on events relating to VM creation (compute.instance.create.* events) and are distinguished by request id. This will create short streams which comprise of events relating to VM creation generated by a specific request – it could be a stream comprising of a compute.instance.create.start followed by a compute.instance.create.end event which relate to the specific request.<\/p>\n
In the example below, the stream is initiated by any event matching compute.instance.create.* which has a unique request_id; any subsequent events which have this event_type and the same request_id are put into the stream and the stream is terminated when the fire_criteria are met – either a compute.instance.create.end or compute.instance.create.error message are obtained for the request_id.<\/p>\n
- name: create_instance_trigger\r\n  debug_level: 2\r\n  distinguished_by:\r\n- request_id\r\n  expiration: \"$last + 1h\"\r\n  fire_pipeline : \"instance_create_start\"\r\n  expire_pipeline: \"instance_create_end\"\r\n  match_criteria:\r\n- event_type:\r\n- compute.instance.create.*\r\n  fire_criteria:\r\n    - event_type:\r\n      - compute.instance.create.end\r\n      - compute.instance.create.error\r\n<\/pre>\n
Streams are captured via triggers as shown above: triggers are defined in winchester\/triggers.yaml within stacktach-sandbox directory.<\/p>\n
Web Application<\/strong><\/p>\n
So now that we have a solution which tracks activity on our Openstack cluster and generates appropriate streams, we can build a small application which can give some insight into the operation of our resources. We built a small web-based application which tracks the time taken to perform standard API calls (eg launch VM, delete VM, snapshot VM) on the cluster. Stacktach was configured to generate streams relating to the operations we were interested in and the web application queries the stream database generated by Stacktach. The web tool allows us to see how these operations vary with time (daily, weekly, monthly) averaged over the entire node and broken down by flavor and node. The short video below shows the tool in action.<\/p>\n
https:\/\/youtu.be\/e4aCYLK2RkI<\/a><\/p>\n
Schlagw\u00f6rter: cloud performance<\/a>, monitoring<\/a>, notifications<\/a>, openstack<\/a>, Stacktach<\/a>
<\/div>","protected":false},"excerpt":{"rendered":"
In one of our projects, FICORE the continuation of FIWARE, we need to offer an Openstack-based service. One aspect of service operations is to understand the performance of the system and one particular aspect of this is to understand how long basic operations take; it is interesting to see how this evolves over time as, […]<\/p>\n","protected":false},"author":101,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[1],"tags":[749,225,748,240,747],"features":[],"class_list":["post-9648","post","type-post","status-publish","format-standard","hentry","category-allgemein","tag-cloud-performance","tag-monitoring","tag-notifications","tag-openstack","tag-stacktach"],"yoast_head":"\nA Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System - Service Engineering (ICCLab & SPLab)<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System\" \/>\n<meta property=\"og:description\" content=\"In one of our projects, FICORE the continuation of FIWARE, we need to offer an Openstack-based service. One aspect of service operations is to understand the performance of the system and one particular aspect of this is to understand how long basic operations take; it is interesting to see how this evolves over time as, […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/\" \/>\n<meta property=\"og:site_name\" content=\"Service Engineering (ICCLab & SPLab)\" \/>\n<meta property=\"article:published_time\" content=\"2016-01-26T12:37:23+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-01-26T15:12:43+00:00\" \/>\n<meta name=\"author\" content=\"Bruno Grazioli\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Bruno Grazioli\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"7 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/\"},\"author\":{\"name\":\"Bruno Grazioli\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/769a2455bfa3cbcc87857218f19abeba\"},\"headline\":\"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System\",\"datePublished\":\"2016-01-26T12:37:23+00:00\",\"dateModified\":\"2016-01-26T15:12:43+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/\"},\"wordCount\":1152,\"commentCount\":2,\"keywords\":[\"cloud performance\",\"monitoring\",\"notifications\",\"openstack\",\"Stacktach\"],\"articleSection\":[\"*.*\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/\",\"name\":\"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System - Service Engineering (ICCLab & SPLab)\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\"},\"datePublished\":\"2016-01-26T12:37:23+00:00\",\"dateModified\":\"2016-01-26T15:12:43+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/769a2455bfa3cbcc87857218f19abeba\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/icclab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/\",\"name\":\"Service Engineering (ICCLab & SPLab)\",\"description\":\"A Blog of the ZHAW Zurich University of Applied Sciences\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/769a2455bfa3cbcc87857218f19abeba\",\"name\":\"Bruno Grazioli\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/07d475be0415d5914aa49599b3295e4d972f971e46f5d7ab89d474327ab7b5f0?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/07d475be0415d5914aa49599b3295e4d972f971e46f5d7ab89d474327ab7b5f0?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/07d475be0415d5914aa49599b3295e4d972f971e46f5d7ab89d474327ab7b5f0?s=96&d=mm&r=g\",\"caption\":\"Bruno Grazioli\"},\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/author\/gaea\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System - Service Engineering (ICCLab & SPLab)","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/","og_locale":"en_US","og_type":"article","og_title":"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System","og_description":"In one of our projects, FICORE the continuation of FIWARE, we need to offer an Openstack-based service. One aspect of service operations is to understand the performance of the system and one particular aspect of this is to understand how long basic operations take; it is interesting to see how this evolves over time as, […]","og_url":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/","og_site_name":"Service Engineering (ICCLab & SPLab)","article_published_time":"2016-01-26T12:37:23+00:00","article_modified_time":"2016-01-26T15:12:43+00:00","author":"Bruno Grazioli","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Bruno Grazioli","Est. reading time":"7 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/"},"author":{"name":"Bruno Grazioli","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/769a2455bfa3cbcc87857218f19abeba"},"headline":"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System","datePublished":"2016-01-26T12:37:23+00:00","dateModified":"2016-01-26T15:12:43+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/"},"wordCount":1152,"commentCount":2,"keywords":["cloud performance","monitoring","notifications","openstack","Stacktach"],"articleSection":["*.*"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/","url":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/","name":"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System - Service Engineering (ICCLab & SPLab)","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#website"},"datePublished":"2016-01-26T12:37:23+00:00","dateModified":"2016-01-26T15:12:43+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/769a2455bfa3cbcc87857218f19abeba"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/icclab\/a-tool-for-understanding-openstack-cloud-performance-using-stacktach-and-the-openstack-notification-system\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/icclab\/"},{"@type":"ListItem","position":2,"name":"A Tool for Understanding OpenStack Cloud Performance using Stacktach and the OpenStack Notification System"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/icclab\/#website","url":"https:\/\/blog.zhaw.ch\/icclab\/","name":"Service Engineering (ICCLab & SPLab)","description":"A Blog of the ZHAW Zurich University of Applied Sciences","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/769a2455bfa3cbcc87857218f19abeba","name":"Bruno Grazioli","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/07d475be0415d5914aa49599b3295e4d972f971e46f5d7ab89d474327ab7b5f0?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/07d475be0415d5914aa49599b3295e4d972f971e46f5d7ab89d474327ab7b5f0?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/07d475be0415d5914aa49599b3295e4d972f971e46f5d7ab89d474327ab7b5f0?s=96&d=mm&r=g","caption":"Bruno Grazioli"},"url":"https:\/\/blog.zhaw.ch\/icclab\/author\/gaea\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/9648","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/users\/101"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/comments?post=9648"}],"version-history":[{"count":10,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/9648\/revisions"}],"predecessor-version":[{"id":9658,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/9648\/revisions\/9658"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/media?parent=9648"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/categories?post=9648"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/tags?post=9648"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/features?post=9648"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}