In two previous blog posts – here and here – we discussed our experience with deploying OpenWhisk on Kubernetes on OpenStack. As applied researchers at the Service Prototyping Lab, we are investigating potential use cases for such setups and for FaaS-based applications in general. In this blog post, we will therefore describe how we built a sample MQTT-based application that shows OpenWhisk in action for sensor data processing for future Internet of Things and smart dust scenarios.
The basic idea of the application is that it consumes data from an MQTT feed, stores it in a database and provides a means to access the database via a web UI. The architecture of the application is shown in the figure below. The application is based on this blog post.
As can be seen from the figure, the motion sensor publishes messages to a specific topic. The trigger service provider subscribes to the same topic and fires OpenWhisk triggers that invoke the action
addReadingToDb that inserts each message to a CouchDB database. A user can point a browser at the
getLatestTenReadings web action every 10 seconds to refresh the page.
The IoT motion sensor and the MQTT broker were provided by our friends at the Cloud Accounting and Billing Initiative and were simply based on a Raspberry Pi with an attached motion sensor publishing to an MQTT broker. The motion sensor publishes a message to a specific topic (
lexxito in this case) every time it detects motion. The simple message contains a JSON object with just two keys: time and status.
We needed a persistent service that managed subscriptions from OpenWhisk trigger creations to topics and fired a trigger every time a new message was received. We used the service provider from the blog post mentioned above. However, we had to make some modifications to it as it assumed Cloudant – the IBM proprietary data store – as a backend. As we are using vanilla OpenWhisk, we needed an alternative and we decided to use CouchDB instead. The modified code can be found here.(We also found and fixed a bug with the
remove_trigger function in the
feed_controller.js file which threw an error every time we tried to delete a trigger).
We then deployed a CouchDB instance on our Kubernetes cluster in a separate namespace using the Helm package manager following these instructions. Next, we had to configure the CouchDB database so that it can be used by the service provider. To do that, we created a database in our CouchDB instance with the name
topic_listeners by running the following:
$ curl -X PUT "http://username:password@COUCHDB_HOST:COUCHDB_PORT/topic_listeners"
This call returns the very informative response:
Which indicates that the configuration of the database was successful.
Then, we needed to add some views to our database. Views in CouchDB are used to query and filter documents. We created a file called
views.json and added it to CouchDB by running the following command:
$ curl -X PUT "http://username:password@COUCHDB_HOST:COUCHDB_PORT/topic_listeners/_design/subscriptions" --data-binary @views.json
We had to create the
readings database that will be used by the
addReadingToDb action on CouchDB by running the following command:
$ curl -X PUT "http://username:password@COUCHDB_HOST:COUCHDB_PORT/readings"
Then, we were ready to deploy the service provider on our Kubernetes cluster by containerizing it first using the provided Dockerfile, pushing the docker container to Docker Hub, and then creating a
deployment.yaml file for it.
Then, we deployed the service provider by running:
$ kubectl apply -f deployment.yaml
And to expose the pod as a service, we ran:
$ kubectl expose pod mqtt-provider --port='3000' --type='NodePort'
With that, the service provider was now ready to accept trigger registrations.
We then turned our attention to configuring OpenWhisk to connect to the service provider in order to register triggers with it. To do that, we created a shared package under the
/whisk.system namespace to make it accessible to all users, with the name
mqtt by running the following:
$ wsk package create \ --shared yes \ -p provider_endpoint "http://SERVICE_PROVIDER_HOST:SERVICE_PROVIDER_NODEPORT/mqtt" mqtt
And we added some description to the package:
$ wsk package update mqtt \ -a description 'The mqtt package provides functionality to connect to MQTT brokers'
Next, we created a feed action and associated it with the
mqtt package. A feed action is an action that manages the lifecycle of triggers. The code for the feed action used in our sample application can be found here.
The feed action will take care of creating and removing triggers. This means that when a user would like to create a trigger, i.e. subscribe to a specific topic on an MQTT broker and fire triggers whenever a new message is published on the topic, the feed action will send a request to the service provider in order to register the trigger with the specific topic on the MQTT broker.
We added the feed action to the
mqtt package by running the following command:
$ wsk action create -a feed true mqtt/mqtt_feed feed_action.js
Once we had the feed action configured, we were able to associate a trigger with the
lexxito topic on the MQTT broker. To do that, we ran the following command:
$ wsk trigger create /guest/feed_trigger \ --feed /whisk.system/mqtt/mqtt_feed \ -p topic 'lexxito' \ -p url 'mqtt://mqtt.demonstrator.info' \ -p authKey $WHISK_GUEST_AUTH \ -p triggerName '/guest/feed_trigger'
Next, we created the action
addReadingToDb that will insert each message to the CouchDB database. The code for the action can be viewed here.
addReadingToDb action is a NodeJS package that used the
sync-request NPM module to perform a POST request to the readings database on CouchDB. This was achieved by following the instructions from this blog post on how to package NPM dependencies for OpenWhisk actions.
Next, we added the
AddReadingToDb action to OpenWhisk by running the following command:
$ wsk action create addReadingToDb \ --kind nodejs:default addReadingToDb.zip
Next, we had to create an OpenWhisk rule that connected the
feed_trigger to the
addReadingToDb action so that every time the
feed_trigger is fired, the
addReadingToDb action gets invoked. An OpenWhisk rule associates a trigger with an action, invoking the action when the associated trigger is fired.
To create the rule, we ran the following:
$ wsk rule create mqttRule /guest/feed_trigger addReadingToDb
Now every time a new message is published on the topic
lexxito, a trigger is fired by the trigger service provider carrying with it the contents of the message to the
addReadingToDb action. The action will then be invoked and will add the contents of the message to the
We then turned our attention to the user-facing component of our sample application.
We wanted to serve static content within OpenWhisk without using any external services like GitHub Pages, for example. Since OpenWhisk doesn’t provide a convenient way to serve static content, we looked into using OpenWhisk web actions to return a HTML page.
An OpenWhisk web action is an action that’s accessible via REST API without needing credentials nor the OpenWhisk CLI. That is, upon the creation of a web action, a URL is created for it that can be hit to invoke the web action.
We created a web action,
renderHtml.js that returns a HTML page to show the latest readings from the
readings database. The code for the
renderHtml.js action can be viewed here.
To create the
renderHtml.js web action, we ran the following command:
$ wsk action create renderHtml renderHtml.js --web true
Within the HTML code that
getLatestTenReadings, every ten seconds in order to refresh the page. The
getLatestTenReadings web action queries the
readings database and returns the ten most-recent entries in the
The code for
getLatestReadings can be viewed here.
Similar to the
addReadingToDb action, this is a NodeJS package that uses the
sync-request NPM module to retrieve the ten most-recent entries from the readings database.
We created the
getLatestTenReadings web action by running the following command:
$ wsk action create getLatestTenReadings \ --kind nodejs:default getLatestTenReadings.zip \ --web true
And we got the URL to
getLatestTenReadings by running the following:
$ wsk action get getLatestTenReadings --url
We got the URL to the
renderHtml web action by running the following:
$ wsk action get renderHtml --url
Finally, by pointing the browser at this endpoint, the dynamic MQTT data should be visible.
Getting a basic application running on OpenWhisk that uses a custom external feed, requires database support, and serves static content is not a trivial task. Perhaps some of the issues that we faced are addressed in commercial variants of OpenWhisk, but the developer experience of getting a simple app operational on vanilla OpenWhisk is not so straightforward for what is touted to be a very developer-oriented platform.
Nonetheless, OpenWhisk remains an exciting platform with a lot of potential. But, from our experience, there is still too much friction associated with getting a basic application running.