Operating system process management has traditionally been an activity at the system level. An init system starts processes at boot time, at random events, and at the user’s (well, root’s) request. Well-known init systems are systemd, openrc and the venerable sysvinit. Without the boot-time component, supervisor processes such as supervisord, runit, daemontools or r6 form another related set of applications with emphasis on automated restart which we have covered already in previous blog posts.
In PaaS environments, multi-tenancy is crucial and process managers need to support it beyond just showing the users owning the processes. While multiple tenants can be perfectly mapped to multiple system users, and information from these users can be aggregated with automation tools, the permission management can quickly become complex and, first and foremost, already requires root permissions to create new users. Therefore, we have explored in the context of the cloud robotics research initiative how to manage processes purely as an ordinary user, but still for multiple tenants. A real need to do so is the fact that by running applications on ROS, dozens of ROS nodes get spawned quickly, many spawning further subprocesses and, due to early terminations, additionally causing fully detached and zombie processes which, when also changing their process group id, do not share a relation anymore to the original application.
The following excerpt from a list of processes, captured from a real application running on top of ROS, shows the dilemma. There are many processes. It is not immediately clear which one was invoked by the user (as the root could be salt-minion instead of bash) and on whose behalf, and it is not clear which ones are system processes which were there already or have been invoked independently from the application (note the nested use of roslaunch, for instance).
We have therefore designed UProcman, the user-friendly and user-level process manager. The figure below shows how it essentially works. Processes are launched through uprocman either with or without tenant labelling. Calling the tool without tenant label causes full visibility over both the default tenant and all other tenants which are kept in a local database which, like all state information by the tool, is conveniently kept as local hidden files. When running processes are queried, the process hierarchy including children and disconnected former children (now adopted by init) is shown. When processes are stopped eventually, all related processes are forced to terminate as well, and accounting information is written which can later be scanned for multi-tenant rating, charging and billing.
The following screenshots show a simple run-through with the most important invocation modes of uprocman. The first command shows how to launch a process. Note how the parameter -T is specified which enables full tracing (through strace). This maximises the chance to catch all dependent processes, at a small or large runtime overhead cost. For our use case, the network on the robot is the limiting factor and the tracing seems to not cause any slowdown as the heavy processing is anyway shifted to the cloud. For other cases, one heuristic solution requiring further research would be to maintain blacklists or whitelists on which processes with known behaviour to trace or not to trace so that CPU-intensive non-forking processes can be run at full speed.
Next, the status of the launched process is shown. It has born a child process but is otherwise not very interesting.
The tracing machinery becomes visible by calling status with -T as well. It now shows both the management processes which handle the tracing and the zombie status (Z) of the sleep process. Which, as one might guess, is not strictly zombie in operating systems parlance, but the term has been sticking somehow.
Another command in uprocman is supervise which would continuously monitor the processes for termination to get precise accounting information. Or, rather, semi-precise, because notifications about process terminations on the /proc filesystem are not implemented, and thus polling needs to be used. Alternatively, the processes can be manually terminated as shown on the next picture.
The value of using uprocman becomes apparent when revisiting the complex ROS nodes example given in the introduction. The last screenshot shows the full list of processes launched inadvertently by launching the mapping functionality on a turtlebot which coordinates the LIDAR laser scanner, the wheels, the map creation, and dozens of other helper processes.
UProcman is still proof-of-concept code to explore how to run applications provided by different users on user- or time-shared resource-constrained devices. As usual, you can grab the code from the Git repository and play with the provided testapp-* scripts which show different behaviour as well as with more complex long-running applications.