{"id":9967,"date":"2016-04-27T17:40:59","date_gmt":"2016-04-27T15:40:59","guid":{"rendered":"https:\/\/blog.zhaw.ch\/icclab\/?p=9967"},"modified":"2021-03-04T18:51:26","modified_gmt":"2021-03-04T16:51:26","slug":"challenges-with-running-ros-on-kubernetes","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/challenges-with-running-ros-on-kubernetes\/","title":{"rendered":"Challenges with running ROS on Kubernetes"},"content":{"rendered":"

The goal of the Cloud Robotics initiative<\/a> of the SPLab<\/a> is to ease the integration of Cloud Computing and Robotics workloads. One of the first things we need to sort out is how to leverage different networking models available on the cloud to support these mixed workloads.<\/p>\n

In this blog post we’ll see one little handy trick to have ROS nodes<\/strong> run as pods (and services) in any Kubernetes cluster<\/strong> so that they can transparently communicate using a ROS topic.<\/p>\n

<\/p>\n

We’re using the Robot Operating System (ROS) <\/a>on both robots and the cloud. As you\u00a0 (might not) know, ROS 1 uses a ROS master to act as registrar of other ROS nodes that act as publishers<\/strong> or subscribers<\/strong> on a topic<\/strong>. When a publisher creates a topic, and a subscriber subscribes to it, what actually happens is that the master informs the publisher, so that it\u00a0 directly sends unicast messages to the subscriber when there’s something to say (a pretty low-level pub\/sub implementation). See figure below (credits Clearpath Robotics, for more info click here<\/a>).<\/p>\n

<\/p>\n

One of the assumptions of the figure above is that publishers and subscribers can directly address each other. ROS has some very specific networking requirements<\/a> for topic messages to be relayed correctly.<\/p>\n

Now, we want to deploy the same thing on a Kubernetes cluster running in the cloud, but, following the best practices of current cloud development, we’re packaging each ROS node in a separated Docker container. Consequently, we’re going to use a separate pod for each ROS node<\/strong> so as to allow nodes to be running on any cluster server. Also, we’d be very happy not to need any specific network setup<\/strong> and use Kubernetes overlay networking<\/a> to do the work for us.<\/p>\n

Kubernetes has a nifty discovery mechanism based on DNS<\/a> that lets your pods find services<\/strong><\/a> by name<\/strong> rather than meddling with IPs. This is a very important feature in a cloud where the assumption is that pods might disappear any time, but services are entities that might exist independently of the supporting implementation in a pod.<\/p>\n

According to the Kubernetes DNS documentation<\/a>, “the only objects to which we are assigning DNS names are Services”, hence if we want our ROS nodes to find each other without implementing a discovery mechanism, we need to create a service for each ROS node we intend to deploy.<\/p>\n

This is quite straightforward for the ROS Master. Here’s the pod specification:<\/p>\n

apiVersion: v1\r\nkind: Pod\r\nmetadata:\r\n\u00a0 name: master\r\n\u00a0 labels:\r\n\u00a0\u00a0\u00a0 name: master\r\nspec:\r\n\u00a0 containers: \r\n\u00a0\u00a0 -\u00a0 name: master\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 image: ros:indigo\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 ports:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - containerPort: 11311\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 name: master\r\n\u00a0\u00a0\u00a0\u00a0\u00a0 args:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 - roscore<\/code><\/pre>\n
No big surprises. We’re using the excellent “official” ROS containers from Docker Hub (https:\/\/hub.docker.com\/_\/ros\/<\/a>) and exposing port 11311 where the ROS master will accept incoming connections from other nodes.<\/p>\n
Here’s the ROS Master service specification:<\/p>\n
apiVersion: v1\r\nkind: Service\r\nmetadata: \r\n  name: master\r\nspec: \r\n  clusterIP: None\r\n  ports: \r\n    - port: 11311\r\n      protocol: TCP\r\n  selector: \r\n    name: master\r\n  type: ClusterIP\r\n<\/code><\/pre>\n
We are creating a service on port 11311 of type “ClusterIP”. This will create a Virtual IP in the Kubernetes network and make sure that all the traffic to that IP and port<\/strong> are forwarded by the Kube-Proxies to the right pod(s).<\/p>\n
How services get a virtual IP and port in Kubernetes. Credits: Kubernetes Introduction – talk given by Daniel Smith at Kubenetes User Group meetup #2 in Mountain View on 4\/21\/2015<\/figcaption><\/figure>\n
However, the problem comes when trying to do the same with the publisher and the subscriber. In fact, if one takes a look at the Python implementation of ROS publishers and subscribers<\/a>, they use dynamic ports<\/strong> and register those as reachable endpoints to the master. Unfortunately Kubernetes services currently do not support dynamic ports (nor port ranges for that matter) and routing would not work.<\/p>\n
Luckily for us, the concept of headless services<\/strong><\/a> lets us:<\/p>\n