{"id":1343,"date":"2020-10-27T14:45:06","date_gmt":"2020-10-27T13:45:06","guid":{"rendered":"http:\/\/blog.zhaw.ch\/splab\/?p=1343"},"modified":"2020-10-27T14:45:06","modified_gmt":"2020-10-27T13:45:06","slug":"processing-iot-sensor-data-with-semi-stateful-cloud-functions","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/splab\/2020\/10\/27\/processing-iot-sensor-data-with-semi-stateful-cloud-functions\/","title":{"rendered":"Processing IoT Sensor Data with Semi-Stateful Cloud Functions"},"content":{"rendered":"\n

At Zurich University of Applied Sciences, we are currently building a test track to link applied teaching with research and innovation. Such a facility allows for covering a whole range of topics: programming, autonomous driving, robotics, cloud, serverless, continuums, sensing, open data, data science, and various computing paradigms. We expect a video to be available around November that explains the facility and especially the teaching element. In this research blog post, we already report on interesting observations around the uplink between sensors and FaaS. We expect these insights to bring benefits to companies building IoT-cloud integrations.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n\n\n\n\n

The (simplified) architecture of the system is shown below. Various types of sensors report events to cloud functions. These functions are typically considered stateless, although in practice they are never completely (see older related post<\/a> on building probabilistically stateful applications). To manage state over a longer period of time, an in-memory database is attached to the function; for security reasons, the access is controlled by a restricted virtual network. As we are still covered by GCP Research Credits for Serverless Data Integration, we are basing this work on the respective GCP offerings: GCF, VPC Connector, and the Redis Memorystore.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The ‘hit’ function will for instance increase a counter anytime a car passes the corresponding sensor. It reads the Redis credentials from environment variables. Furthermore, it reads a secret from the environment too, with a fallback to a variable definition in a secrets file. This is a pragmatic way to protect against damaging unauthorised function calls: Instead of requiring authentication at the invocation level, the function is open for public invocation but any actual processing, for example writing values to Redis, is protected. We regard low-effort starting with technologies important for prototypical research and teaching, and for this reason also disregarded GCP’s IoT Core that came with heavy double authentication (keys + JWT) requirements that are redirecting too much effort into security engineering rather than freeing the effort up for advances in technology. (Of course we assume and demand that anybody who operates such solutions in production would then take care of appropriate security.)<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Once the described system was up and working, the research question emerged: Is Redis really needed? Could we not just store the counter variable as global cloud function variable, hence letting it survive requests?<\/p>\n\n\n\n

There are two potential problems that may interfere with that idea. Function instances (the containers or equivalent isolation units beneath FaaS) are kept around for some time, but not forever. Specifically, an invocation will get a new instance if:<\/p>\n\n\n\n