{"id":10965,"date":"2016-12-14T10:46:49","date_gmt":"2016-12-14T08:46:49","guid":{"rendered":"https:\/\/blog.zhaw.ch\/icclab\/?p=10965"},"modified":"2016-12-14T11:35:27","modified_gmt":"2016-12-14T09:35:27","slug":"on-storage-computation-for-a-serverless-environment","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/","title":{"rendered":"On-storage computation for a serverless environment"},"content":{"rendered":"<p>The <a href=\"http:\/\/martinfowler.com\/articles\/serverless.html\">serverless architecture<\/a>\u00a0is getting a lot of attention and there is a lot of talk going on about it (<a href=\"http:\/\/www.forbes.com\/sites\/janakirammsv\/2016\/10\/12\/why-enterprises-should-care-about-serverless-computing\/#1d874b155555\">forbes<\/a>, <a href=\"https:\/\/gigaom.com\/2016\/11\/10\/serverless-enabled-storage-its-a-big-deal\/\">gigaom<\/a>, <a href=\"http:\/\/techbeacon.com\/aws-lambda-serverless-apps-5-things-you-need-know-about-serverless-computing\">techbeacon<\/a>). This new architecture\u00a0is especially useful for developers since there is no need to worry about deployment or interactions between different servers.\u00a0The developer only needs to worry about the code, a function. Functions are the way applications are written in this architecture, otherwise known as Function as a Service (<a href=\"https:\/\/blog.zhaw.ch\/icclab\/faas-function-hosting-services-and-their-technical-characteristics\/\">FaaS<\/a>).<\/p>\n<p><!--more--><\/p>\n<p>It is very clear that object storage could fit\u00a0the serverless architecture\u00a0to present a possible solution to the problem of stateful functions. Object storage does not make any distinctions between different file types (e.g.: mp3, txt, jpeg) and saves the data and the metadata of an object separately, this is a very important property of object storage. In <a href=\"http:\/\/docs.openstack.org\/developer\/swift\/\">OpenStack Swift<\/a>\u00a0objects are stored in containers whose storage can grow as needed. This is important for object management and that&#8217;s why important for the computation on storage objects.<\/p>\n<p>But how can we bring object storage and FaaS closer so we can begin to explore solving the stateful function problem?<\/p>\n<p>A few years ago Rackspace sponsored\u00a0a project that would allow you to run code directly on your storage node. This project is known as <a href=\"http:\/\/zerovm.org\">zerovm<\/a>, which is\u00a0a promising concept and very relevant for serverless architectures.<\/p>\n<p>Now what is zerovm? Zerovm is a platform that allows to bring computation to the data and not create traffic and bottlenecks by bringing the data to the compute. It is deployed together with OpenStack Swift on a storage node. The thing that zerovm allows us to do now is that a python script can be written and uploaded to the node, zerovm. This script resembles a function in the serverless pattern quite a lot. This function now sits there and only runs when it is invoked. It can be invoked together with a job-description, this job-description is a json file that specifies parameters, storage-container that should be used in the function when executed.<\/p>\n<p>This is exactly what is looked for in the serverless world, on-storage computation that is directly executed where the data is located without creating an overhead from downloading the data to compute nodes. A function that defines what is to be done with the storage object that are given as input, that only is executed when it is invoked.<\/p>\n<p>Here you can see a very basic example of zerovm. Let&#8217;s assume there is a huge logfile stored as an object in a container. We want to go find out how many times the error code &#8220;banana&#8221; has showed up in this file, so we write a short python script <code>demo.py<\/code><\/p>\n<pre><code>i = 0\r\nwith open('\/dev\/input') as fp:\r\n    for line in fp:\r\n        if \"banana\" in line:\r\n            i = i + 1\r\nprint i\r\n<\/code><\/pre>\n<p>This code needs to be packed and uploaded to the containers on which the code should be ran. In this case the container is called <code>demo<\/code>.<br \/>\nWhat is <code>\/dev\/input<\/code>? This can be specified in the job description, so let&#8217;s have a look at the <code>job.json<\/code> file.<\/p>\n<pre><code>[{\r\n    \"name\": \"zerovmdemo\",\r\n    \"exec\": {\r\n        \"path\": \"file:\/\/python2.7:python\",\r\n        \"args\": \"demo.py\"\r\n    },\r\n    \"devices\": [\r\n        {\"name\": \"python2.7\"},\r\n        {\"name\": \"stdout\"},\r\n        {\"name\": \"input\", \"path\": \"swift:\/\/~\/demo\/logile.txt\"},\r\n        {\"name\": \"image\", \"path\": \"swift:\/\/~\/demo\/demo.tar\"}\r\n    ]\r\n}]\r\n<\/code><\/pre>\n<p>The function can now be executed with a <code>curl<\/code> command and return the number of times that &#8220;banana&#8221; is in the object.<\/p>\n<pre>curl -i -X POST -H \"Content-Type: application\/json\" \\\r\n-H \"X-Auth-Token: $OS_AUTH_TOKEN\" -H \"X-Zerovm-Execute: 1.0\" \\\r\n--data-binary @job.json $OS_STORAGE_URL\r\n<\/pre>\n<p>This is how to deploy and run a function with zerovm, quick and easy.<\/p>\n<p>Stay tuned for more posts on serverless architectures here!<\/p>\n<div class=\"pt-sm\">Schlagw\u00f6rter: <a href=\"https:\/\/blog.zhaw.ch\/icclab\/tag\/faas\/\">faas<\/a>, <a href=\"https:\/\/blog.zhaw.ch\/icclab\/tag\/openstack\/\">openstack<\/a>, <a href=\"https:\/\/blog.zhaw.ch\/icclab\/tag\/serverless\/\">serverless<\/a>, <a href=\"https:\/\/blog.zhaw.ch\/icclab\/tag\/swift\/\">swift<\/a>, <a href=\"https:\/\/blog.zhaw.ch\/icclab\/tag\/zerovm\/\">zerovm<\/a><br><\/div>","protected":false},"excerpt":{"rendered":"<p>The serverless architecture\u00a0is getting a lot of attention and there is a lot of talk going on about it (forbes, gigaom, techbeacon). This new architecture\u00a0is especially useful for developers since there is no need to worry about deployment or interactions between different servers.\u00a0The developer only needs to worry about the code, a function. Functions are [&hellip;]<\/p>\n","protected":false},"author":151,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[1],"tags":[820,240,831,453,832],"features":[],"class_list":["post-10965","post","type-post","status-publish","format-standard","hentry","category-allgemein","tag-faas","tag-openstack","tag-serverless","tag-swift","tag-zerovm"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>On-storage computation for a serverless environment - Service Engineering (ICCLab &amp; SPLab)<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"On-storage computation for a serverless environment\" \/>\n<meta property=\"og:description\" content=\"The serverless architecture\u00a0is getting a lot of attention and there is a lot of talk going on about it (forbes, gigaom, techbeacon). This new architecture\u00a0is especially useful for developers since there is no need to worry about deployment or interactions between different servers.\u00a0The developer only needs to worry about the code, a function. Functions are [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/\" \/>\n<meta property=\"og:site_name\" content=\"Service Engineering (ICCLab &amp; SPLab)\" \/>\n<meta property=\"article:published_time\" content=\"2016-12-14T08:46:49+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2016-12-14T09:35:27+00:00\" \/>\n<meta name=\"author\" content=\"anke\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"anke\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/\"},\"author\":{\"name\":\"anke\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/5273d3e0c0aa9893554dde8b6a3ba991\"},\"headline\":\"On-storage computation for a serverless environment\",\"datePublished\":\"2016-12-14T08:46:49+00:00\",\"dateModified\":\"2016-12-14T09:35:27+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/\"},\"wordCount\":540,\"commentCount\":0,\"keywords\":[\"faas\",\"openstack\",\"serverless\",\"swift\",\"zerovm\"],\"articleSection\":[\"*.*\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/\",\"name\":\"On-storage computation for a serverless environment - Service Engineering (ICCLab &amp; SPLab)\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\"},\"datePublished\":\"2016-12-14T08:46:49+00:00\",\"dateModified\":\"2016-12-14T09:35:27+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/5273d3e0c0aa9893554dde8b6a3ba991\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/icclab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"On-storage computation for a serverless environment\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/\",\"name\":\"Service Engineering (ICCLab &amp; SPLab)\",\"description\":\"A Blog of the ZHAW Zurich University of Applied Sciences\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/5273d3e0c0aa9893554dde8b6a3ba991\",\"name\":\"anke\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/ab37351cc12863d0db77de05c086213ce89e2a19a688bf41ad56ef1a61ff365c?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/ab37351cc12863d0db77de05c086213ce89e2a19a688bf41ad56ef1a61ff365c?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/ab37351cc12863d0db77de05c086213ce89e2a19a688bf41ad56ef1a61ff365c?s=96&d=mm&r=g\",\"caption\":\"anke\"},\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/author\/anke\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"On-storage computation for a serverless environment - Service Engineering (ICCLab &amp; SPLab)","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/","og_locale":"en_US","og_type":"article","og_title":"On-storage computation for a serverless environment","og_description":"The serverless architecture\u00a0is getting a lot of attention and there is a lot of talk going on about it (forbes, gigaom, techbeacon). This new architecture\u00a0is especially useful for developers since there is no need to worry about deployment or interactions between different servers.\u00a0The developer only needs to worry about the code, a function. Functions are [&hellip;]","og_url":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/","og_site_name":"Service Engineering (ICCLab &amp; SPLab)","article_published_time":"2016-12-14T08:46:49+00:00","article_modified_time":"2016-12-14T09:35:27+00:00","author":"anke","twitter_card":"summary_large_image","twitter_misc":{"Written by":"anke","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/"},"author":{"name":"anke","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/5273d3e0c0aa9893554dde8b6a3ba991"},"headline":"On-storage computation for a serverless environment","datePublished":"2016-12-14T08:46:49+00:00","dateModified":"2016-12-14T09:35:27+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/"},"wordCount":540,"commentCount":0,"keywords":["faas","openstack","serverless","swift","zerovm"],"articleSection":["*.*"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/","url":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/","name":"On-storage computation for a serverless environment - Service Engineering (ICCLab &amp; SPLab)","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#website"},"datePublished":"2016-12-14T08:46:49+00:00","dateModified":"2016-12-14T09:35:27+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/5273d3e0c0aa9893554dde8b6a3ba991"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/icclab\/on-storage-computation-for-a-serverless-environment\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/icclab\/"},{"@type":"ListItem","position":2,"name":"On-storage computation for a serverless environment"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/icclab\/#website","url":"https:\/\/blog.zhaw.ch\/icclab\/","name":"Service Engineering (ICCLab &amp; SPLab)","description":"A Blog of the ZHAW Zurich University of Applied Sciences","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/5273d3e0c0aa9893554dde8b6a3ba991","name":"anke","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/ab37351cc12863d0db77de05c086213ce89e2a19a688bf41ad56ef1a61ff365c?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/ab37351cc12863d0db77de05c086213ce89e2a19a688bf41ad56ef1a61ff365c?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/ab37351cc12863d0db77de05c086213ce89e2a19a688bf41ad56ef1a61ff365c?s=96&d=mm&r=g","caption":"anke"},"url":"https:\/\/blog.zhaw.ch\/icclab\/author\/anke\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/10965","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/users\/151"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/comments?post=10965"}],"version-history":[{"count":9,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/10965\/revisions"}],"predecessor-version":[{"id":11019,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/10965\/revisions\/11019"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/media?parent=10965"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/categories?post=10965"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/tags?post=10965"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/features?post=10965"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}