{"id":1312,"date":"2020-09-26T06:04:27","date_gmt":"2020-09-26T04:04:27","guid":{"rendered":"http:\/\/blog.zhaw.ch\/splab\/?p=1312"},"modified":"2020-09-26T06:04:27","modified_gmt":"2020-09-26T04:04:27","slug":"smart-cloud-log-file-compression","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/","title":{"rendered":"Smart Cloud Log File Compression"},"content":{"rendered":"\n

Software developers and service operators need log files to identify issues, detect anomalies and trace execution behaviour. The amount of generated log data is increasing<\/a>, and often log files need to be kept for longer periods of time due to regulations. To preserve logs in a cost-efficient manner, they are typically compressed, at certain cost for running the compression, and then stored in long-term archives, again at certain cost per size-duration products<\/a>. The goal is decrease both cost components, but there are certain trade-offs, for instance a highly efficient compression that consumes a lot of CPU but leads to better compression ratios, consuming less storage capacity as a result. The decision which compression tools and parameters to use is usually hardcoded<\/a>. We present a smart knowledge-based advisor service to query goal-based adaptive compression commands to maximise savings.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n\n\n\n\n

Compression tooling knowledge can be thought of as a graph containing file formats, standards, compression\/decompression tools and further tools to handle the compressed files, for instance in text searching. Not only the tools, but also their characteristics, retrieved through representative benchmarks, should be captured so that the best compression method can be searched for any given situation. An excerpt of such a graph is shown below along with some explanations.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Precise knowledge on compression is important. Of key interest is the relation between ratio and timing – we often want fast and best compression combined, although in practice, we might have to make a compromise for one of them. Also, there are some restrictions, e.g. some of the best compression tools might have a 2GB file size limit. The duration\/ratio trade-offs are shown below, along with the “pareto front” that prevents us from reaching absolute optimums.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Having access to such a graph and the compression duration\/memory\/ratio\/limits information is the first step towards a holistic solution. A second step is then the logic to determine the best tool and its parameters for a given goal, e.g. “best compression ratio” or “fastest decompression time (for searching)”. The third step is to make this functionality available as an API, to align with various cloud service models. Our system, compressadvisor<\/strong>, along with the compressgraph<\/strong> model and the automated benchmarking is shown below.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The compressadvisor is simple to use through a RESTful API along with a provided CLI tool so that e.g. scripts can always query the advisor instead of hardcoding the execution commands.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

We provide a demo instance<\/a> of the advisor running at Google App Engine along with an upcoming code\/data\/benchmark repository. The innovative log management solution will be presented at the CCCI 2020 conference<\/a> in November where the open source repository will be launched as well. The work is based upon work supported by Google Cloud, with GCP Research Credits for Serverless Data Integration, and furthermore linked to other log management activities supported by an innovation starting grant of the Swiss Leading House MENA.<\/p>\n

Schlagw\u00f6rter: benchmark<\/a>, compression<\/a>, graph<\/a>, leading house<\/a>, log management<\/a>, logging<\/a>, service<\/a>
<\/div>","protected":false},"excerpt":{"rendered":"

Software developers and service operators need log files to identify issues, detect anomalies and trace execution behaviour. The amount of generated log data is increasing, and often log files need to be kept for longer periods of time due to regulations. To preserve logs in a cost-efficient manner, they are typically compressed, at certain cost […]<\/p>\n","protected":false},"author":203,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[4],"tags":[164,97,118,132,159,145,163],"features":[],"class_list":["post-1312","post","type-post","status-publish","format-standard","hentry","category-research","tag-benchmark","tag-compression","tag-graph","tag-leading-house","tag-log-management","tag-logging","tag-service"],"yoast_head":"\nSmart Cloud Log File Compression - Service Prototyping Lab<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Smart Cloud Log File Compression\" \/>\n<meta property=\"og:description\" content=\"Software developers and service operators need log files to identify issues, detect anomalies and trace execution behaviour. The amount of generated log data is increasing, and often log files need to be kept for longer periods of time due to regulations. To preserve logs in a cost-efficient manner, they are typically compressed, at certain cost […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/\" \/>\n<meta property=\"og:site_name\" content=\"Service Prototyping Lab\" \/>\n<meta property=\"article:published_time\" content=\"2020-09-26T04:04:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg\" \/>\n<meta name=\"author\" content=\"Josef Spillner\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Josef Spillner\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/\"},\"author\":{\"name\":\"Josef Spillner\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/33a5227c6cc1fa3329279d33c3cc440e\"},\"headline\":\"Smart Cloud Log File Compression\",\"datePublished\":\"2020-09-26T04:04:27+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/\"},\"wordCount\":492,\"commentCount\":0,\"image\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg\",\"keywords\":[\"benchmark\",\"compression\",\"graph\",\"leading house\",\"log management\",\"logging\",\"service\"],\"articleSection\":[\"Research\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/\",\"url\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/\",\"name\":\"Smart Cloud Log File Compression - Service Prototyping Lab\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg\",\"datePublished\":\"2020-09-26T04:04:27+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/33a5227c6cc1fa3329279d33c3cc440e\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#primaryimage\",\"url\":\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg\",\"contentUrl\":\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/splab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Smart Cloud Log File Compression\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/splab\/\",\"name\":\"Service Prototyping Lab\",\"description\":\"A Blog of the ZHAW Zurich University of Applied Sciences\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/splab\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/33a5227c6cc1fa3329279d33c3cc440e\",\"name\":\"Josef Spillner\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/c57d7019e91373902c08eac230e003cb4f8498ecf68c9d119e9ecb82fdf6c400?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c57d7019e91373902c08eac230e003cb4f8498ecf68c9d119e9ecb82fdf6c400?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c57d7019e91373902c08eac230e003cb4f8498ecf68c9d119e9ecb82fdf6c400?s=96&d=mm&r=g\",\"caption\":\"Josef Spillner\"},\"description\":\"Head of the Service Prototyping Lab. Research on distributed systems and application computing paradigms. Senior lecturer at Zurich University of Applied Sciences.\",\"url\":\"https:\/\/blog.zhaw.ch\/splab\/author\/spio\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Smart Cloud Log File Compression - Service Prototyping Lab","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/","og_locale":"en_GB","og_type":"article","og_title":"Smart Cloud Log File Compression","og_description":"Software developers and service operators need log files to identify issues, detect anomalies and trace execution behaviour. The amount of generated log data is increasing, and often log files need to be kept for longer periods of time due to regulations. To preserve logs in a cost-efficient manner, they are typically compressed, at certain cost […]","og_url":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/","og_site_name":"Service Prototyping Lab","article_published_time":"2020-09-26T04:04:27+00:00","og_image":[{"url":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg","type":"","width":"","height":""}],"author":"Josef Spillner","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Josef Spillner","Estimated reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/"},"author":{"name":"Josef Spillner","@id":"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/33a5227c6cc1fa3329279d33c3cc440e"},"headline":"Smart Cloud Log File Compression","datePublished":"2020-09-26T04:04:27+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/"},"wordCount":492,"commentCount":0,"image":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg","keywords":["benchmark","compression","graph","leading house","log management","logging","service"],"articleSection":["Research"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/","url":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/","name":"Smart Cloud Log File Compression - Service Prototyping Lab","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/splab\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#primaryimage"},"image":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg","datePublished":"2020-09-26T04:04:27+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/33a5227c6cc1fa3329279d33c3cc440e"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#primaryimage","url":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg","contentUrl":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/09\/logman.jpg"},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/09\/26\/smart-cloud-log-file-compression\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/splab\/"},{"@type":"ListItem","position":2,"name":"Smart Cloud Log File Compression"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/splab\/#website","url":"https:\/\/blog.zhaw.ch\/splab\/","name":"Service Prototyping Lab","description":"A Blog of the ZHAW Zurich University of Applied Sciences","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/splab\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/33a5227c6cc1fa3329279d33c3cc440e","name":"Josef Spillner","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/secure.gravatar.com\/avatar\/c57d7019e91373902c08eac230e003cb4f8498ecf68c9d119e9ecb82fdf6c400?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c57d7019e91373902c08eac230e003cb4f8498ecf68c9d119e9ecb82fdf6c400?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c57d7019e91373902c08eac230e003cb4f8498ecf68c9d119e9ecb82fdf6c400?s=96&d=mm&r=g","caption":"Josef Spillner"},"description":"Head of the Service Prototyping Lab. Research on distributed systems and application computing paradigms. Senior lecturer at Zurich University of Applied Sciences.","url":"https:\/\/blog.zhaw.ch\/splab\/author\/spio\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/posts\/1312","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/users\/203"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/comments?post=1312"}],"version-history":[{"count":1,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/posts\/1312\/revisions"}],"predecessor-version":[{"id":1318,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/posts\/1312\/revisions\/1318"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/media?parent=1312"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/categories?post=1312"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/tags?post=1312"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/features?post=1312"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}