{"id":1112,"date":"2019-12-20T18:24:02","date_gmt":"2019-12-20T17:24:02","guid":{"rendered":"http:\/\/blog.zhaw.ch\/splab\/?p=1112"},"modified":"2019-12-20T19:42:37","modified_gmt":"2019-12-20T18:42:37","slug":"building-a-singer-io-tap-for-an-open-data-source","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/splab\/2019\/12\/20\/building-a-singer-io-tap-for-an-open-data-source\/","title":{"rendered":"Building a Singer.io tap for an open data source"},"content":{"rendered":"\n

Singer.io<\/a> is an open-source JSON-based data shifting (ETL: extract, transform, load) framework, designed to bring simplicity when moving data between a source and a destination service on the Internet. In this post, we present the framework as entry point into the world of SaaS-level data exchange and some associated research questions.<\/p>\n\n\n\n\n\n\n\n

A script that extracts data from a source (like Stripe payments<\/a>) and outputs it following the Singer format convention is called a tap<\/strong>. A script that consumes that output and works with it (like putting it into a .csv file<\/a> or a PostgreSQL<\/a> database) is called a target<\/strong>.<\/p>\n\n\n\n

The idea is that, using the defined convention based on JSON data exchange, any tap can be connected to any target. With this in mind, our goal is to see how easy it is to create a tap and connect it to an existing target, as a first step in the evaluation of this framework for prototyping SaaS-level applications working with diverse data sources. In this context, Singer is placed on a higher level compared to general data orchestration frameworks for cloud environments such as Alluxio<\/a>, and links up with many multi-cloud and cross-cloud ideas but is also subject to the same issues such as the least common denominator problem<\/a>.<\/p>\n\n\n\n

Although there are a lot of already made taps<\/a> to download and use, those are useful only if you already have meaningful data in any of those sources. For our testing purposes, that was simply not the case.<\/p>\n\n\n\n

So today we’re going to build a simple Python tap that takes data from an open data source and outputs it using the Singer spec.<\/p>\n\n\n\n

Creating a tap from the template<\/h2>\n\n\n\n

In order to develop a tap, we first need to install the Singer library:<\/p>\n\n\n\n

pip install singer-python<\/code><\/pre>\n\n\n\n
Next, we’re installing cookiecutter<\/em> and downloading the provided tap template, to have something to start with:<\/p>\n\n\n\n
pip install cookiecutter\ncookiecutter https:\/\/github.com\/singer-io\/singer-tap-template.git\nproject_name [e.g. 'tap-facebook']: tap-weather\npackage_name [tap_weather]:<\/code><\/pre>\n\n\n\n
After downloading the template, cookiecutter<\/em> will ask for a project name. We chose tap-weather<\/strong>, because our data source will be weather related. Then a package name is required- pressing enter will use the suggested name.<\/p>\n\n\n\n
We now have a blank canvas with a few useful functions to get started with. To follow the singer spec, a tap must output two different types of messages:<\/p>\n\n\n\n