Building a Sample MQTT-based Application on OpenWhisk

In two previous blog posts – here and here – we discussed our experience with deploying OpenWhisk on Kubernetes on OpenStack. As applied researchers at the Service Prototyping Lab, we are investigating potential use cases for such setups and for FaaS-based applications in general. In this blog post, we will therefore describe how we built a sample MQTT-based application that shows OpenWhisk in action for sensor data processing for future Internet of Things and smart dust scenarios.

The basic idea of the application is that it consumes data from an MQTT feed, stores it in a database and provides a means to access the database via a web UI. The architecture of the application is shown in the figure below. The application is based on this blog post.

As can be seen from the figure, the motion sensor publishes messages to a specific topic. The trigger service provider subscribes to the same topic and fires OpenWhisk triggers that invoke the action addReadingToDb that inserts each message to a CouchDB database. A user can point a browser at the renderHtml.js web action which returns a HTML page to show the latest readings. Within the page, there’s a JavaScript function that invokes the getLatestTenReadings web action every 10 seconds to refresh the page.

The IoT motion sensor and the MQTT broker were provided by our friends at the Cloud Accounting and Billing Initiative and were simply based on a Raspberry Pi with an attached motion sensor publishing to an MQTT broker. The motion sensor publishes a message to a specific topic ( lexxito in this case) every time it detects motion. The simple message contains a JSON object with just two keys: time and status.

We needed a persistent service that managed subscriptions from OpenWhisk trigger creations to topics and fired a trigger every time a new message was received. We used the service provider from the blog post mentioned above. However, we had to make some modifications to it as it assumed  Cloudant – the IBM proprietary data store – as a backend. As we are using vanilla OpenWhisk, we needed an alternative and we decided to use CouchDB instead. The modified code can be found here.(We also found and fixed a bug with the remove_trigger function in the feed_controller.js file which threw an error every time we tried to delete a trigger).

We then deployed a CouchDB instance on our Kubernetes cluster in a separate namespace using the Helm package manager following these instructions. Next, we had to configure the CouchDB database so that it can be used by the service provider. To do that, we created a database in our CouchDB instance with the name topic_listeners by running the following:

$ curl -X PUT "http://username:password@COUCHDB_HOST:COUCHDB_PORT/topic_listeners"

This call returns the very informative response:

{"ok":true}

Which indicates that the configuration of the database was successful.

Then, we needed to add some views to our database. Views in CouchDB are used to query and filter documents. We created a file called views.json and added it to CouchDB by running the following command:

$ curl -X PUT "http://username:password@COUCHDB_HOST:COUCHDB_PORT/topic_listeners/_design/subscriptions" --data-binary @views.json

We had to create the readings database that will be used by the addReadingToDb action on CouchDB by running the following command:

$ curl -X PUT "http://username:password@COUCHDB_HOST:COUCHDB_PORT/readings"

Then, we were ready to deploy the service provider on our Kubernetes cluster by containerizing it first using the provided Dockerfile, pushing the docker container to Docker Hub, and then creating a deployment.yaml file for it.

Then, we deployed the service provider by running:

$ kubectl apply -f deployment.yaml

And to expose the pod as a service, we ran:

$ kubectl expose pod mqtt-provider --port='3000' --type='NodePort'

With that, the service provider was now ready to accept trigger registrations.

We then turned our attention to configuring OpenWhisk to connect to the service provider in order to register triggers with it. To do that, we created a shared package under the /whisk.system namespace to make it accessible to all users, with the name mqtt by running the following:

$ wsk package create \
--shared yes \
-p provider_endpoint "http://SERVICE_PROVIDER_HOST:SERVICE_PROVIDER_NODEPORT/mqtt" mqtt

And we added some description to the package:

$ wsk package update mqtt \
-a description 'The mqtt package provides functionality to connect to MQTT brokers'

Next, we created a feed action and associated it with the mqtt package. A feed action is an action that manages the lifecycle of triggers. The code for the feed action used in our sample application can be found here.

The feed action will take care of creating and removing triggers. This means that when a user would like to create a trigger, i.e. subscribe to a specific topic on an MQTT broker and fire triggers whenever a new message is published on the topic, the feed action will send a request to the service provider in order to register the trigger with the specific topic on the MQTT broker.

We added the feed action to the mqtt package by running the following command:

$ wsk action create -a feed true mqtt/mqtt_feed feed_action.js

Once we had the feed action configured, we were able to associate a trigger with the lexxito topic on the MQTT broker. To do that, we ran the following command:

$ wsk trigger create /guest/feed_trigger \
--feed /whisk.system/mqtt/mqtt_feed \
-p topic 'lexxito' \
-p url 'mqtt://mqtt.demonstrator.info' \
-p authKey $WHISK_GUEST_AUTH \
-p triggerName '/guest/feed_trigger'

Next, we created the action addReadingToDb that will insert each message to the CouchDB database. The code for the action can be viewed here.

The addReadingToDb action is a NodeJS package that used the sync-request NPM module to perform a POST request to the readings database on CouchDB. This was achieved by following the instructions from this blog post on how to package NPM dependencies for OpenWhisk actions.

Next, we added the AddReadingToDb action to OpenWhisk by running the following command:

$ wsk action create addReadingToDb \
--kind nodejs:default addReadingToDb.zip

Next, we had to create an OpenWhisk rule that connected the feed_trigger to the addReadingToDb action so that every time the feed_trigger is fired, the addReadingToDb action gets invoked. An OpenWhisk rule associates a trigger with an action, invoking the action when the associated trigger is fired.

To create the rule, we ran the following:

$ wsk rule create mqttRule /guest/feed_trigger addReadingToDb

Now every time a new message is published on the topic lexxito, a trigger is fired by the trigger service provider carrying with it the contents of the message to the addReadingToDb action. The action will then be invoked and will add the contents of the message to the readings database.

The UI

We then turned our attention to the user-facing component of our sample application.

We wanted to serve static content within OpenWhisk without using any external services like GitHub Pages, for example. Since OpenWhisk doesn’t provide a convenient way to serve static content, we looked into using OpenWhisk web actions to return a HTML page.

An OpenWhisk web action is an action that’s accessible via REST API without needing credentials nor the OpenWhisk CLI. That is, upon the creation of a web action, a URL is created for it that can be hit to invoke the web action.

We created a web action, renderHtml.js that returns a HTML page to show the latest readings from the readings database. The code for the renderHtml.js action can be viewed here.

To create the renderHtml.js web action, we ran the following command:

$ wsk action create renderHtml renderHtml.js --web true

Within the HTML code that renderHtml.js returns, there’s a JavaScript script that invokes another web action, getLatestTenReadings, every ten seconds in order to refresh the page. The getLatestTenReadings web action queries the readings database and returns the ten most-recent entries in the readings database.

The code for getLatestReadings can be viewed here.

Similar to the addReadingToDb action, this is a NodeJS package that uses the sync-request NPM module to retrieve the ten most-recent entries from the readings database.

We created the getLatestTenReadings web action by running the following command:

$ wsk action create getLatestTenReadings \
--kind nodejs:default getLatestTenReadings.zip \
--web true

And we got the URL to getLatestTenReadings by running the following:

$ wsk action get getLatestTenReadings --url

We got the URL to the renderHtml web action by running the following:

$ wsk action get renderHtml --url

Finally, by pointing the browser at this endpoint, the dynamic MQTT data should be visible.

Lessons Learnt

Getting a basic application running on OpenWhisk that uses a custom external feed, requires database support, and serves static content is not a trivial task. Perhaps some of the issues that we faced are addressed in commercial variants of OpenWhisk, but the developer experience of getting a simple app operational on vanilla OpenWhisk is not so straightforward for what is touted to be a very developer-oriented platform.

Nonetheless, OpenWhisk remains an exciting platform with a lot of potential. But, from our experience, there is still too much friction associated with getting a basic application running.