{"id":10565,"date":"2016-09-02T14:41:49","date_gmt":"2016-09-02T12:41:49","guid":{"rendered":"https:\/\/blog.zhaw.ch\/icclab\/?p=10565"},"modified":"2019-08-05T14:35:27","modified_gmt":"2019-08-05T12:35:27","slug":"benchmarking-cloud-native-database-systems","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/","title":{"rendered":"Benchmarking cloud-native database systems"},"content":{"rendered":"\n

by Josef Spillner<\/a><\/p>\n\n\n

For reliable, user-controlled and trustworthy file storage in the cloud, free software prototypes like NubiSave<\/a> have become great tools to investigate and lift the barrier towards acceptable migration paths. For structured data storage and processing, several approaches to database-as-a-service (DBaaS<\/a>) have been proposed by researchers and developers but a clear recommendation of how to best manage rows or records of data in the cloud from a practicality angle is still absent. Partially, the question about how to do this is due to the different pricing structures and availability guarantees by the providers which are not trivial to compare. Often, running the database system as set of replicated or sharded containers being part of the application appears to be a valid alternative to the binding of existing commercial DBaaS, if done correctly. After all, cloud providers would offer the same technical guarantees for any of their services. An analysis of which configuration works better and is less expensive would thus be needed.<\/p>\n

<\/p>\n

On a more abstract level, this leads into a conflict which we’re going to see a lot more in the future: Should we build DIY solutions on top of IaaS, or should be buy into whatever is offered by the PaaS? This question cannot be answered for all services at once. But right now, it is especially pressuring for data processing applications, and in particular for many applications using a plain database management system.<\/p>\n

CNDBbench<\/strong> to the rescue. This cloud-native database benchmark<\/em> application, while still under development<\/a>, offers a systematic way of checking how to manage data from your cloud application which ideally is already built in a cloud-native way<\/a>.<\/p>\n

The figure below shows the basic working modes of CNDBbench. It builds (using docker build<\/strong>) benchmark containers specific to database interfaces (SQL, JSON) and deploys them in different combinations locally and in the cloud. Either readily provided database services or benchmark-controlled instances (single or clustered) are used. The containers can again be hosted either in the provider’s own container engine-as-a-service, or (due to still many issues with this variant) inside a kubernetes<\/strong> instance on a plain old virtual machine.<\/p>\n

\"Several
Several options to run and benchmark a database system in the cloud.<\/figcaption><\/figure>\n

Currently, CNDBbench supports five relational, document-oriented and hybrid database systems: MySQL<\/a>, PostgreSQL<\/a>, MongoDB<\/a>, CouchDB<\/a> and Crate<\/a>. Furthermore, it supports the equivalent DBaaS interfaces like Aurora<\/a>. To follow a single-source dataset approach, a conversion from the source format into the destination format (as tuples or nested structures) is performed.<\/p>\n

Hence, CNDBbench uses docker-compose<\/strong> to build the composite benchmarking suite, consisting of (1) the specific benchmark container, (2) a read-only reference dataset container, (3) a writeable results container, and (4) optionally the database system container(s). Each check is repeated a number of times to eliminate sporadic outliers. The results are written as portable CSV files which are then plotted into diagrams with gnuplot<\/strong>. Knowledge gained from the graphs includes the absolute times to perform database operations, the relative stability (inverse deviation) of measurements, and the uninterrupted behaviour of the overall system in the presence of (provoked) faults.<\/p>\n

The following figure offers a first glimpse at measurements produced by CNDBbench. Four queries are run which execute in comparable times with most systems but diverge a lot with CouchDB.<\/p>\n

\"Running
Running four queries on locally run database systems in their default docker containers.<\/figcaption><\/figure>\n

You can fetch the code to CNDBbench from its Git repository<\/a> as with all other software created in the Service Prototyping Lab<\/a>. A scientific paper about the system is being written and will be submitted to a matching conference soon.<\/p>

Schlagw\u00f6rter: cloud-native application<\/a>, containers<\/a>, database<\/a>, dbaas<\/a>
<\/div>","protected":false},"excerpt":{"rendered":"

For reliable, user-controlled and trustworthy file storage in the cloud, free software prototypes like NubiSave have become great tools to investigate and lift the barrier towards acceptable migration paths. For structured data storage and processing, several approaches to database-as-a-service (DBaaS) have been proposed by researchers and developers but a clear recommendation of how to best […]<\/p>\n","protected":false},"author":486,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[1],"tags":[558,609,525,804],"features":[],"class_list":["post-10565","post","type-post","status-publish","format-standard","hentry","category-allgemein","tag-cloud-native-application","tag-containers","tag-database","tag-dbaas"],"yoast_head":"\nBenchmarking cloud-native database systems - Service Engineering (ICCLab & SPLab)<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Benchmarking cloud-native database systems\" \/>\n<meta property=\"og:description\" content=\"For reliable, user-controlled and trustworthy file storage in the cloud, free software prototypes like NubiSave have become great tools to investigate and lift the barrier towards acceptable migration paths. For structured data storage and processing, several approaches to database-as-a-service (DBaaS) have been proposed by researchers and developers but a clear recommendation of how to best […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/\" \/>\n<meta property=\"og:site_name\" content=\"Service Engineering (ICCLab & SPLab)\" \/>\n<meta property=\"article:published_time\" content=\"2016-09-02T12:41:49+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2019-08-05T12:35:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png\" \/>\n<meta name=\"author\" content=\"icclab\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"icclab\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/\"},\"author\":{\"name\":\"icclab\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/045c6bde7e681e689e4fc051d8932563\"},\"headline\":\"Benchmarking cloud-native database systems\",\"datePublished\":\"2016-09-02T12:41:49+00:00\",\"dateModified\":\"2019-08-05T12:35:27+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/\"},\"wordCount\":607,\"commentCount\":0,\"image\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png\",\"keywords\":[\"cloud-native application\",\"containers\",\"database\",\"dbaas\"],\"articleSection\":[\"*.*\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/\",\"name\":\"Benchmarking cloud-native database systems - Service Engineering (ICCLab & SPLab)\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png\",\"datePublished\":\"2016-09-02T12:41:49+00:00\",\"dateModified\":\"2019-08-05T12:35:27+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/045c6bde7e681e689e4fc051d8932563\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#primaryimage\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png\",\"contentUrl\":\"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png\",\"width\":990,\"height\":394,\"caption\":\"Several options to run and benchmark a database system in the cloud.\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/icclab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Benchmarking cloud-native database systems\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/\",\"name\":\"Service Engineering (ICCLab & SPLab)\",\"description\":\"A Blog of the ZHAW Zurich University of Applied Sciences\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/045c6bde7e681e689e4fc051d8932563\",\"name\":\"icclab\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/7b13169e03783f50e96b96fa2ff222b9c530d13c3125f077c7c44f729b857a51?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/7b13169e03783f50e96b96fa2ff222b9c530d13c3125f077c7c44f729b857a51?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/7b13169e03783f50e96b96fa2ff222b9c530d13c3125f077c7c44f729b857a51?s=96&d=mm&r=g\",\"caption\":\"icclab\"},\"url\":\"https:\/\/blog.zhaw.ch\/icclab\/author\/icclab\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Benchmarking cloud-native database systems - Service Engineering (ICCLab & SPLab)","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/","og_locale":"en_US","og_type":"article","og_title":"Benchmarking cloud-native database systems","og_description":"For reliable, user-controlled and trustworthy file storage in the cloud, free software prototypes like NubiSave have become great tools to investigate and lift the barrier towards acceptable migration paths. For structured data storage and processing, several approaches to database-as-a-service (DBaaS) have been proposed by researchers and developers but a clear recommendation of how to best […]","og_url":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/","og_site_name":"Service Engineering (ICCLab & SPLab)","article_published_time":"2016-09-02T12:41:49+00:00","article_modified_time":"2019-08-05T12:35:27+00:00","og_image":[{"url":"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png","type":"","width":"","height":""}],"author":"icclab","twitter_card":"summary_large_image","twitter_misc":{"Written by":"icclab","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/"},"author":{"name":"icclab","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/045c6bde7e681e689e4fc051d8932563"},"headline":"Benchmarking cloud-native database systems","datePublished":"2016-09-02T12:41:49+00:00","dateModified":"2019-08-05T12:35:27+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/"},"wordCount":607,"commentCount":0,"image":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png","keywords":["cloud-native application","containers","database","dbaas"],"articleSection":["*.*"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/","url":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/","name":"Benchmarking cloud-native database systems - Service Engineering (ICCLab & SPLab)","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#primaryimage"},"image":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png","datePublished":"2016-09-02T12:41:49+00:00","dateModified":"2019-08-05T12:35:27+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/045c6bde7e681e689e4fc051d8932563"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#primaryimage","url":"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png","contentUrl":"https:\/\/blog.zhaw.ch\/icclab\/files\/2016\/09\/dbaas-options.png","width":990,"height":394,"caption":"Several options to run and benchmark a database system in the cloud."},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/icclab\/benchmarking-cloud-native-database-systems\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/icclab\/"},{"@type":"ListItem","position":2,"name":"Benchmarking cloud-native database systems"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/icclab\/#website","url":"https:\/\/blog.zhaw.ch\/icclab\/","name":"Service Engineering (ICCLab & SPLab)","description":"A Blog of the ZHAW Zurich University of Applied Sciences","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/icclab\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/icclab\/#\/schema\/person\/045c6bde7e681e689e4fc051d8932563","name":"icclab","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/7b13169e03783f50e96b96fa2ff222b9c530d13c3125f077c7c44f729b857a51?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/7b13169e03783f50e96b96fa2ff222b9c530d13c3125f077c7c44f729b857a51?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/7b13169e03783f50e96b96fa2ff222b9c530d13c3125f077c7c44f729b857a51?s=96&d=mm&r=g","caption":"icclab"},"url":"https:\/\/blog.zhaw.ch\/icclab\/author\/icclab\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/10565","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/users\/486"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/comments?post=10565"}],"version-history":[{"count":5,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/10565\/revisions"}],"predecessor-version":[{"id":12516,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/posts\/10565\/revisions\/12516"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/media?parent=10565"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/categories?post=10565"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/tags?post=10565"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/icclab\/wp-json\/wp\/v2\/features?post=10565"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}