{"id":1192,"date":"2020-03-18T07:03:30","date_gmt":"2020-03-18T06:03:30","guid":{"rendered":"http:\/\/blog.zhaw.ch\/splab\/?p=1192"},"modified":"2020-03-18T07:03:30","modified_gmt":"2020-03-18T06:03:30","slug":"deploying-singer-io-to-the-cloud","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/","title":{"rendered":"Deploying Singer.io to the Cloud"},"content":{"rendered":"\n<p>As presented in a <a href=\"https:\/\/blog.zhaw.ch\/splab\/2019\/12\/20\/building-a-singer-io-tap-for-an-open-data-source\">prior post<\/a>, Singer.io is a modern, open-source ETL (Extract, Transform and Load) framework for integrating data from various sources, including online datasets and services, with a focus on being simple and light-weight.  The basics of the framework were explored in our last post on the topic, so we will refer you to that if you are unfamiliar.<\/p>\n\n\n\n<p>This post is about our process for deploying Singer to the cloud, more specifically, to the <a href=\"https:\/\/www.cloudfoundry.org\/\">Cloud Foundry<\/a> open source cloud application platform. This was done in the context of researching the maturity of data transformation tools in a cloud-native environment.  We will explore the options for deploying Singer taps and targets to a cloud provider and discuss our implementation and deployment process in detail.<\/p>\n\n\n\n<!--more-->\n\n\n\n<h2 class=\"wp-block-heading\">Exploring our options<\/h2>\n\n\n\n<p>Several ideas were considered to deploy Singer tap-target pairs to the cloud. We quickly found out two options that did not work, and those are:<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Simply deploying the tap and target as separate apps with no changes<\/li><li>Deploying the whole pair as one app<\/li><\/ul>\n\n\n\n<p>The background for why this second option was not viable has a fairly straightforward explanation. Singer is a specification. It merely specifies the message format that the tap and target use in their communication, thus allowing taps to communicate with any target and vice versa. The way this is implemented is for a tap&#8217;s standard output to be piped into a target&#8217;s standard input.<\/p>\n\n\n\n<p>Additionally, the maintenance state of the different taps and targets varies, with some instances of conflicting dependencies not being unheard of, which <a href=\"https:\/\/github.com\/singer-io\/getting-started\/blob\/master\/docs\/RUNNING_AND_DEVELOPING.md\">Singer themselves recommend solving by running each half of the pipeline in a separate Python virtual environment<\/a>.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Options we considered<\/h2>\n\n\n\n<p>The first obvious solution was to create a Docker container with the tap and target pair, giving us the freedom to run the pair and the needed virtual environments. The drawbacks of this method were that we effectively lose the flexibility of Singer by binding the tap with the target.<\/p>\n\n\n\n<p>The second attempt had to be more invasive to the code of Singer taps and targets, as we decided to implement cloud-native versions of these apps that communicate via the network. To better match Singer&#8217;s format, <a href=\"https:\/\/grpc.io\/\">gRPC<\/a> was the method we selected.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Modifying Singer to use gRPC<\/h2>\n\n\n\n<p>We will use the <a href=\"https:\/\/www.singer.io\/tap\/exchange-rates-api\/\">exchange rates API tap<\/a> and the <a href=\"https:\/\/www.singer.io\/target\/csv\/\">CSV target<\/a> as a starting point, due to their simplicity in terms of code. To set up our gRPC-Singer, we needed a prototype file to define our service:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>package singer;\n\n\/\/ Interface exported by the server.\nservice Singer {\n  \/\/ A client-to-server streaming RPC.\n  rpc Sing(stream Message) returns (Status) {}\n}\nmessage Message {\n  string blob = 1;\n}\n\nmessage Status {\n  string status = 1;\n}<\/code><\/pre>\n\n\n\n<p>We chose string as the message format as Singer targets already parse messages from strings. The status message is not that important to us.<\/p>\n\n\n\n<p>As can be seen by the service definition, the Sing RPC method receives a stream of messages. The next step is to generate the Python code we need to implement this service. The command is as follows:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>python -m grpc_tools.protoc -I. --python_out=. --grpc_python_out=. .\/singer.proto<\/code><\/pre>\n\n\n\n<p>Now is the time to modify the code to perform the gRPC, starting from the tap. Here we need to transform the function that writes the messages to standard output, to a Python generator. To do this we first need to replace two of Singer&#8217;s utility functions:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>def write_record(stream_name, record, stream_alias=None, time_extracted=None):\n    return singer_pb2.Message(blob=json.dumps(singer.RecordMessage(stream=(stream_alias or stream_name),\n                                record=record,\n                                time_extracted=time_extracted).asdict()))\n\n\ndef write_schema(stream_name, schema, key_properties, bookmark_properties=None, stream_alias=None):\n    if isinstance(key_properties, (str, bytes)):\n        key_properties = [key_properties]\n    if not isinstance(key_properties, list):\n        raise Exception(\"key_properties must be a string or list of strings\")\n\n    return singer_pb2.Message(blob=json.dumps(singer.SchemaMessage(\n            stream=(stream_alias or stream_name),\n            schema=schema,\n            key_properties=key_properties,\n            bookmark_properties=bookmark_properties).asdict()))<\/code><\/pre>\n\n\n\n<p>To make the function a generator, we need it to yield these messages:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>...\nyield write_schema('exchange_rate', schema, 'date')\n...\nyield write_record('exchange_rate', record)\n...<\/code><\/pre>\n\n\n\n<p>Finally, we need the code to make it an <a href=\"https:\/\/docs.aiohttp.org\/en\/stable\/\">aiohttp<\/a> server to receive our request, and the code that actually makes the RPC call.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>@routes.post('\/')\nasync def main(request):\n    params = await request.json()\n    config = params['config']\n    target = params['target']\n\n    state = {}\n\n    start_date = state.get('start_date') or config.get('start_date') or datetime.utcnow().strftime(DATE_FORMAT)\n    start_date = singer.utils.strptime_with_tz(start_date).date().strftime(DATE_FORMAT)\n    with grpc.insecure_channel(f'{target}.apps.internal:8080') as channel:\n            stub = singer_pb2_grpc.SingerStub(channel)\n            response = do_sync(config.get('base', 'USD'), start_date)\n            status = stub.Sing(response)\n    return web.json_response(status.status)<\/code><\/pre>\n\n\n\n<p>Now on to the target. Turns out the modifications here are less intrusive, but we still need to tell the target that the actual Singer message is now a property of the message we are sending in our stream:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>message = message.blob<\/code><\/pre>\n\n\n\n<p>And of course, we need to set up our gRPC server:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>class SingerServicer(singer_pb2_grpc.SingerServicer):\n\n    def Sing(self, request_iterator, context):\n        config = {}\n        persist_messages(config.get('delimiter', ','),\n                                 config.get('quotechar', '\"'),\n                                 request_iterator,\n                                 config.get('destination_path', ''))\n        return singer_pb2.Status(status = \"OK\")\n\n\ndef serve():\n    server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))\n    singer_pb2_grpc.add_SingerServicer_to_server(\n        SingerServicer(), server)\n    server.add_insecure_port('[::]:8080')\n    server.start()\n    server.wait_for_termination()\n\n\nif __name__ == '__main__':\n    logging.basicConfig()\n    serve()<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Deploying to Cloud Foundry<\/h2>\n\n\n\n<p>Deploying these apps to Cloud Foundry could have been very straightforward, but we wanted our taps and targets to share Singer&#8217;s philosophy of being generic. Unfortunately, that means any new taps need to follow the service definition we just wrote, but at least we can make them work with any target. For the case of Cloud Foundry, we can achieve this with <a href=\"https:\/\/docs.cloudfoundry.org\/concepts\/understand-cf-networking.html\">Container to Container Networking<\/a>. <\/p>\n\n\n\n<p>An internal route to the target needs to be set up:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cf map-route singergrpctarget apps.internal --hostname singergrpctarget<\/code><\/pre>\n\n\n\n<p>In addition, network policies that allow the tap to reach the target need to be created.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>cf add-network-policy singergrpctap --destination-app singergrpctarget --port 8080 --protocol tcp<\/code><\/pre>\n\n\n\n<p>With this, the tap can resolve the target at <code>singergrpctarget.apps.internal<\/code>.<\/p>\n\n\n\n<p>Running traditional Singer the user is the only one aware of the combination of tap and target used, but in our case, the tap needs to know where to send the gRPC call. For this purpose, the request to the target needs to have the target, and the configuration for the tap.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>{\n    \"target\": \"singergrpctarget\",\n    \"config\": {\n        \"base\": \"CHF\",\n        \"start_date\": \"2020-02-20\"\n    }\n}<\/code><\/pre>\n\n\n\n<p>Because of our internal networking, the tap knows that it can resolve the target at <code>{target}.apps.internal<\/code>.<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"530\" src=\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png\" alt=\"Cloud Foundry dashboard view of the gRPC tap receiving the sample request.\" class=\"wp-image-1226\" srcset=\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png 1024w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-300x155.png 300w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-768x398.png 768w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1536x796.png 1536w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-676x350.png 676w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25.png 1720w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>The Singer gRPC tap receiving the above request<\/figcaption><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"439\" src=\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-39-06-1024x439.png\" alt=\"Cloud Foundry dashboard view of the gRPC target.\" class=\"wp-image-1227\" srcset=\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-39-06-1024x439.png 1024w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-39-06-300x129.png 300w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-39-06-768x329.png 768w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-39-06-1536x658.png 1536w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-39-06-676x290.png 676w, https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-39-06.png 1694w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><figcaption>Our gRPC target in the Cloud Foundry dashboard<\/figcaption><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">Closing thoughts<\/h2>\n\n\n\n<p>Deploying Singer.io to the cloud was not straightforward, and the lack of a traditional comprehensive API meant that a generic wrapper could not be created trivially, but we have demonstrated a possible method to convert taps and targets to a more cloud-friendly format and hope to inspire more usage of the framework in a cloud-native context.<\/p>\n\n\n\n<p>Our experiences with Singer and Cloud Foundry also prompted an interesting observation, that while local (CLI) applications can very easily exchange information by <a href=\"https:\/\/www.linuxjournal.com\/article\/2156\">being piped into each other<\/a>, the same cannot be said about PaaS applications. While several solutions, including our gRPC implementation or fully integrated frameworks that negate the need for a data flow, are possible, there is still development cost associated with implementing a data pipeline between different applications, an aspect inviting future work in streamlining the process. <\/p>\n<div class=\"pt-sm\">Schlagw\u00f6rter: <a href=\"http:\/\/blog.zhaw.ch\/splab\/tag\/cloud-foundry\/\">cloud foundry<\/a>, <a href=\"http:\/\/blog.zhaw.ch\/splab\/tag\/cloud-native\/\">cloud-native<\/a>, <a href=\"http:\/\/blog.zhaw.ch\/splab\/tag\/etl\/\">etl<\/a>, <a href=\"http:\/\/blog.zhaw.ch\/splab\/tag\/paas\/\">paas<\/a>, <a href=\"http:\/\/blog.zhaw.ch\/splab\/tag\/python\/\">python<\/a>, <a href=\"http:\/\/blog.zhaw.ch\/splab\/tag\/singer\/\">singer<\/a><br><\/div>","protected":false},"excerpt":{"rendered":"<p>As presented in a prior post, Singer.io is a modern, open-source ETL (Extract, Transform and Load) framework for integrating data from various sources, including online datasets and services, with a focus on being simple and light-weight. The basics of the framework were explored in our last post on the topic, so we will refer you [&hellip;]<\/p>\n","protected":false},"author":438,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[4],"tags":[143,20,138,87,121,137],"features":[],"class_list":["post-1192","post","type-post","status-publish","format-standard","hentry","category-research","tag-cloud-foundry","tag-cloud-native","tag-etl","tag-paas","tag-python","tag-singer"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.2 (Yoast SEO v27.2) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>Deploying Singer.io to the Cloud - Service Prototyping Lab<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/\" \/>\n<meta property=\"og:locale\" content=\"en_GB\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Deploying Singer.io to the Cloud\" \/>\n<meta property=\"og:description\" content=\"As presented in a prior post, Singer.io is a modern, open-source ETL (Extract, Transform and Load) framework for integrating data from various sources, including online datasets and services, with a focus on being simple and light-weight. The basics of the framework were explored in our last post on the topic, so we will refer you [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/\" \/>\n<meta property=\"og:site_name\" content=\"Service Prototyping Lab\" \/>\n<meta property=\"article:published_time\" content=\"2020-03-18T06:03:30+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png\" \/>\n<meta name=\"author\" content=\"Panagiotis Gkikopoulos\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Panagiotis Gkikopoulos\" \/>\n\t<meta name=\"twitter:label2\" content=\"Estimated reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/\"},\"author\":{\"name\":\"Panagiotis Gkikopoulos\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/6a362b2982b686a44a453206cb3cee11\"},\"headline\":\"Deploying Singer.io to the Cloud\",\"datePublished\":\"2020-03-18T06:03:30+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/\"},\"wordCount\":949,\"commentCount\":2,\"image\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png\",\"keywords\":[\"cloud foundry\",\"cloud-native\",\"etl\",\"paas\",\"python\",\"singer\"],\"articleSection\":[\"Research\"],\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/\",\"url\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/\",\"name\":\"Deploying Singer.io to the Cloud - Service Prototyping Lab\",\"isPartOf\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png\",\"datePublished\":\"2020-03-18T06:03:30+00:00\",\"author\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/6a362b2982b686a44a453206cb3cee11\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#breadcrumb\"},\"inLanguage\":\"en-GB\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#primaryimage\",\"url\":\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png\",\"contentUrl\":\"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Startseite\",\"item\":\"https:\/\/blog.zhaw.ch\/splab\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Deploying Singer.io to the Cloud\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#website\",\"url\":\"https:\/\/blog.zhaw.ch\/splab\/\",\"name\":\"Service Prototyping Lab\",\"description\":\"A Blog of the ZHAW Zurich University of Applied Sciences\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.zhaw.ch\/splab\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-GB\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/6a362b2982b686a44a453206cb3cee11\",\"name\":\"Panagiotis Gkikopoulos\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-GB\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/88f238fea03fa1a0fe3ee06ef74194f1b2f1ebde2deecb56364cf02b2eb18ec2?s=96&d=mm&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/88f238fea03fa1a0fe3ee06ef74194f1b2f1ebde2deecb56364cf02b2eb18ec2?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/88f238fea03fa1a0fe3ee06ef74194f1b2f1ebde2deecb56364cf02b2eb18ec2?s=96&d=mm&r=g\",\"caption\":\"Panagiotis Gkikopoulos\"},\"url\":\"https:\/\/blog.zhaw.ch\/splab\/author\/pang\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Deploying Singer.io to the Cloud - Service Prototyping Lab","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/","og_locale":"en_GB","og_type":"article","og_title":"Deploying Singer.io to the Cloud","og_description":"As presented in a prior post, Singer.io is a modern, open-source ETL (Extract, Transform and Load) framework for integrating data from various sources, including online datasets and services, with a focus on being simple and light-weight. The basics of the framework were explored in our last post on the topic, so we will refer you [&hellip;]","og_url":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/","og_site_name":"Service Prototyping Lab","article_published_time":"2020-03-18T06:03:30+00:00","og_image":[{"url":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png","type":"","width":"","height":""}],"author":"Panagiotis Gkikopoulos","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Panagiotis Gkikopoulos","Estimated reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#article","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/"},"author":{"name":"Panagiotis Gkikopoulos","@id":"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/6a362b2982b686a44a453206cb3cee11"},"headline":"Deploying Singer.io to the Cloud","datePublished":"2020-03-18T06:03:30+00:00","mainEntityOfPage":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/"},"wordCount":949,"commentCount":2,"image":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png","keywords":["cloud foundry","cloud-native","etl","paas","python","singer"],"articleSection":["Research"],"inLanguage":"en-GB","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/","url":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/","name":"Deploying Singer.io to the Cloud - Service Prototyping Lab","isPartOf":{"@id":"https:\/\/blog.zhaw.ch\/splab\/#website"},"primaryImageOfPage":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#primaryimage"},"image":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#primaryimage"},"thumbnailUrl":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png","datePublished":"2020-03-18T06:03:30+00:00","author":{"@id":"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/6a362b2982b686a44a453206cb3cee11"},"breadcrumb":{"@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#breadcrumb"},"inLanguage":"en-GB","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/"]}]},{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#primaryimage","url":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png","contentUrl":"https:\/\/blog.zhaw.ch\/splab\/files\/2020\/03\/Screenshot-from-2020-03-16-15-38-25-1024x530.png"},{"@type":"BreadcrumbList","@id":"https:\/\/blog.zhaw.ch\/splab\/2020\/03\/18\/deploying-singer-io-to-the-cloud\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Startseite","item":"https:\/\/blog.zhaw.ch\/splab\/"},{"@type":"ListItem","position":2,"name":"Deploying Singer.io to the Cloud"}]},{"@type":"WebSite","@id":"https:\/\/blog.zhaw.ch\/splab\/#website","url":"https:\/\/blog.zhaw.ch\/splab\/","name":"Service Prototyping Lab","description":"A Blog of the ZHAW Zurich University of Applied Sciences","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.zhaw.ch\/splab\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-GB"},{"@type":"Person","@id":"https:\/\/blog.zhaw.ch\/splab\/#\/schema\/person\/6a362b2982b686a44a453206cb3cee11","name":"Panagiotis Gkikopoulos","image":{"@type":"ImageObject","inLanguage":"en-GB","@id":"https:\/\/secure.gravatar.com\/avatar\/88f238fea03fa1a0fe3ee06ef74194f1b2f1ebde2deecb56364cf02b2eb18ec2?s=96&d=mm&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/88f238fea03fa1a0fe3ee06ef74194f1b2f1ebde2deecb56364cf02b2eb18ec2?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/88f238fea03fa1a0fe3ee06ef74194f1b2f1ebde2deecb56364cf02b2eb18ec2?s=96&d=mm&r=g","caption":"Panagiotis Gkikopoulos"},"url":"https:\/\/blog.zhaw.ch\/splab\/author\/pang\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/posts\/1192","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/users\/438"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/comments?post=1192"}],"version-history":[{"count":9,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/posts\/1192\/revisions"}],"predecessor-version":[{"id":1231,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/posts\/1192\/revisions\/1231"}],"wp:attachment":[{"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/media?parent=1192"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/categories?post=1192"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/tags?post=1192"},{"taxonomy":"features","embeddable":true,"href":"https:\/\/blog.zhaw.ch\/splab\/wp-json\/wp\/v2\/features?post=1192"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}