{"id":8245,"date":"2015-06-09T13:54:33","date_gmt":"2015-06-09T11:54:33","guid":{"rendered":"http:\/\/blog.zhaw.ch\/icclab\/?p=8245"},"modified":"2015-06-09T14:21:22","modified_gmt":"2015-06-09T12:21:22","slug":"process-management-in-docker-containers","status":"publish","type":"post","link":"https:\/\/blog.zhaw.ch\/icclab\/process-management-in-docker-containers\/","title":{"rendered":"Process Management in Docker Containers"},"content":{"rendered":"

In the context of the Cloud Native Applications<\/a> (CNA) Seed Project <\/a>we are working on migrating an open source CRM application into to the cloud. After enabling horizontal scalabilty of the original application and moving it onto our OpenStack cloud with the help of CoreOS (etcd, fleet) and docker we’ve now just finished adding the monitoring \/ logging \/ log-collection functionality\u00a0– a blog post describing this process in its detail will follow – which is needed for the next part of the project: enabling automatic scaling<\/em>. As part of this process we’ve learnt some lessons concerning the process management in docker containers which we’d like to share in this post.<\/p>\n

<\/p>\n

Containers and Processes<\/h1>\n

If you use Docker to run an application (e.g a webserver) you just have to write a Dockerfile that installs the webserver and define a command that tells docker what process should be started if the docker container is run. There is no problem yet because if you stop the container the process is stopped, too. If the process in the container exits the container also stops. So there is something like a 1:1 mapping: one container starts\/stops one process. Docker containers are intended to be used like that.<\/p>\n

Pseudo-Dockerfile of a webserver:<\/p>\n

# Use base image\r\nFROM ubuntu:14.04\r\n\r\n# install the service (e.g webserver)\r\nRUN apt-get update && apt-get install webserver\r\n\r\n# add some configuration\/application\r\nADD webserver_config.cfg \/etc\/webserver\/webserver_config.cfg\r\nADD webserver_app \/app\r\n\r\n# process executed when container starts (starting webserver)\r\nCMD \/bin\/start-webserver --foreground<\/pre>\n
Problems arise<\/h1>\n
But very soon you start adding more things to your container. Very likely you start more processes inside the container. An easy way to achieve this is to use a bash-script that is used to start all the processes you use:<\/p>\n
Pseudo-Dockerfile:<\/p>\n
[...]\r\n<\/i>\r\n# install the service (e.g webserver)\r\nRUN apt-get update && apt-get install webserver\r\n\r\n# install more needed applications\r\napt-get install debugging-tool monitoring-tool service-discovery\r\n\r\n[...]\r\n\r\n<\/i>ADD start.sh \/start.sh\r\n\r\n# process executed when container starts (start-script)\r\nCMD \/start.sh<\/pre>\n
Start-script (start.sh):<\/p>\n
#!\/bin\/bash\r\n\/bin\/monitoring-tool &\r\n\/bin\/service-discovery &\r\n\/bin\/start-webserver &\r\n\/bin\/debbugging-tool &\r\ntail -f \/var\/log\/webserver_logfile.log<\/pre>\n
This works well although you might notice that the last line of the script tailing a logfile is a bit hacky and lets you feel a bit uncomfortable. This line is needed to keep the script running and prevents that it exits. The script should not terminate because this is the process started when you run the docker container. If you omit the last line, the container would just stop right after starting all the processes. All these processes will not be running anymore because the container they\u2019re living in is stopped!<\/p>\n
Another problem is logging. With this simple script only the log of the webserver is written to stdout. That means that if you issue the \u2018docker logs\u2019 command, only the webservers logfile will be shown. But it would be much more comfortable if you could read all the logs from all the processes inside the container with this command.<\/p>\n
A typical use case is running an application together with dynamic configuration management. That means the application is running inside the container together with another process which is responsible that the service is running with the latest configuration. This could be some tool like confd<\/i> that watches some keys in a distributed key-value store like etcd<\/i>. If a change of the desired configuration is detected, the tool has to take care of restarting the service (e.g apache) inside the container. If the service is the process the container is started with, it is not possible to restart this process because stopping this process from within the container stops the container.<\/p>\n
You could setup your apache as daemon using init systems like Upstart\/Systemd. This way you solve one and introduce even more problems because you would use the container like a virtual machine. What would happen if the apache service in the container fails? The container would still be running and you wouldn\u2019t notice that the apache process isn\u2019t running anymore.<\/p>\n
Using process supervision – supervisord<\/h1>\n
The solution for this is relatively easy. Just use an init process for the container that handles the lifecycle of the other processes you want to run in the container. Applications that can do this are called process supervision tools<\/i> – Tools like s6<\/a>, runit <\/a>or supervisord<\/a>.<\/p>\n
All you need to do now is to install one of these tools in the docker image, write the configuration which is basically a definition what processes you want to start and finally set the supervision tool as init process in the Dockerfile.<\/p>\n
For Supervisord this can be done as follows:<\/p>\n
Install supervisord (in Ubuntu based image):
\nDockerfile:<\/p>\n
[...]\r\n<\/i>RUN apt-get install -yq supervisor\r\n[...]<\/i><\/pre>\n
Write the configuration (\/etc\/supervisor\/conf.d\/webserver_supervisord.conf):<\/p>\n
[program:webserver]\r\ncommand=\/bin\/start-webserver\r\nnumprocs=1\r\nstdout_logfile=\/dev\/fd\/1\r\nstdout_logfile_maxbytes=0\r\n\r\n[program:monitoring-tool]\r\ncommand=\/bin\/start-monitoring\r\nnumprocs=1\r\nstdout_logfile=\/dev\/fd\/1\r\nstdout_logfile_maxbytes=0\r\n\r\n[program:service-discovery]\r\ncommand=\/bin\/start-servicediscovery\r\nnumprocs=1\r\nstdout_logfile=\/dev\/fd\/1\r\nstdout_logfile_maxbytes=0\r\n\r\n[program:debugging-tool]\r\ncommand=\/bin\/start-debugging\r\nnumprocs=1\r\nstdout_logfile=\/dev\/fd\/1\r\nstdout_logfile_maxbytes=0<\/pre>\n
Set supervisor as init process:
\nDockerfile:<\/p>\n
[...]\r\n<\/i>CMD \/usr\/bin\/supervisord\r\n[...]<\/i><\/pre>\n
Supervisord will be started as the initial process when launching the container. Supervisord then starts and supervises the processes you defined in the configuration file. The stdout_logfile lines in the configuration tells supervisord to log to stdout. That means all outputs from all processes in the container can then be shown by \u2018docker logs\u2019. There are many more options offered by supervisord. For example it can be configured to restart a process if it fails.<\/p>\n
From within the container you can now talk to supervisord with the supervisorctl client. This way you can easily start\/stop\/restart processes defined in the configuration file. This is very similar to Upstart\/Systemd. For example to start the debugging-tool the command would be:<\/p>\n
$ supervisorctl start debugging-tool<\/pre>\n
The eternal process<\/h1>\n
There is still a problem with this approach. Supervisord is always running no matter what other processes are running by supervisord. For your container life cycle that means that as long as supervisord is running your container is running. This maybe is something you don\u2019t want or is something that can confuse other applications monitoring the containers\u2019 status as kind of health management.<\/p>\n
For instance if you use fleet <\/a>to start docker containers in a cluster environment this can really break the expected behavior. A fleet service can be automatically restarted if it fails. If you start a container as fleet service, it would be restarted if the container stops. But with supervisord the container would run forever, even if the webserver inside the container had crashed. Fortunately there is a possibility to shutdown supervisor if a process exits, but this includes writing an additional script and extending the configuration with an event listener.<\/p>\n
The event listener is called when the process specified enters the specified state. In this example, if the webserver process enters the process state fatal, the kill_supervisor.py script is executed.<\/p>\n
Configuration file (\/etc\/supervisor\/conf.d\/webserver_supervisord.conf):<\/p>\n
[...]\r\n<\/i>[eventlistener:webserver_exit]\r\ncommand=\/usr\/bin\/kill_supervisor.py\r\nprocess_name=webserver\r\nevents=PROCESS_STATE_FATAL\r\n[...]<\/i><\/pre>\n
kill_supervisor.py script:<\/p>\n
#!\/usr\/bin\/env python\r\nimport sys\r\nimport os\r\nimport signal\r\n\r\ndef write_stdout(s):\r\n\u00a0\u00a0\u00a0sys.stdout.write(s)\r\n\u00a0\u00a0\u00a0sys.stdout.flush()\r\ndef write_stderr(s):\r\n\u00a0\u00a0\u00a0sys.stderr.write(s)\r\n\u00a0\u00a0\u00a0sys.stderr.flush()\r\ndef main():\r\n\u00a0\u00a0\u00a0while 1:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0write_stdout('READY\\n')\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0line = sys.stdin.readline()\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0write_stdout('This line kills supervisor: ' + line);\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0try:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0pidfile = open('\/var\/run\/supervisord.pid','r')\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0pid = int(pidfile.readline());\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0os.kill(pid, signal.SIGQUIT)\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0except Exception as e:\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0write_stdout('Could not kill supervisor: ' + e.strerror + '\\n')\r\n\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0write_stdout('RESULT 2\\nOK')\r\nif __name__ == '__main__':\r\n\u00a0\u00a0\u00a0main()\r\n\u00a0\u00a0\u00a0import sys<\/pre>\n
Conclusion<\/h1>\n
Process supervision tools like supervisord simplify process handling in docker containers.\u00a0Especially if you need to run multiple processes or need to restart processes\u00a0inside containers.<\/p>\n
Supervisord is one of these process supervision tools. It is easy to install, configure and provides many features. More advanced tasks such as stopping the container as soon as a supervised process is stopped involve writing additional scripts for event handling.<\/p>\n
Word of caution: Supervisord does not resolve all problems regarding process management in docker containers: A problem that is still present is the PID 1 zombie reaping problem<\/a>. Usually an init system is responsible to cleanup zombie processes. Supervisord does not do this. To tackle this problem, you can use one of the other supervision tools mentioned above or use a proper base-image<\/a> for this.<\/p>\n
Sources<\/h1>\n