Docker images have become the valuta franca in the cloud and container platform world. Although on the path to vendor-neutral standardisation (e.g. with OCI also being in Docker Hub for a year now), developers for now have settled on plain Docker as de-facto standard due to the vast ecosystem of base images and dependency images which speed up the rapid prototyping of complex scalable applications. From a production-grade DevOps perspective, a key concern is then to be assured that the containers used are of high quality, not infected by security vulnerabilities, and still containing the latest features available. In this blog post, a novel approach to visualise the situation around a particular container image is presented.
Existing containers are used in Container-as-a-Service (CaaS) platforms such as Kubernetes, OpenShift and Podman explicitly as lowest-level execution unit. Furthermore, they are also used as base images (in Dockerfiles) and as dependency images (in deployment descriptors, templates and compose files). Finally, containers are also transparently auto-built or auto-finalised (with volume mounts) in several Function-as-a-Service (FaaS) and Platform-as-a-Service (PaaS) environments. Any quality or security issue with any container image may quickly invalidate any assumption about the stability and reliability of the container-based applications and services. Thus, better tools to check and represent the characteristics of container images are needed.
For early development stages, there are validation tools such as Hadolint and Dockerlint to find programming issues in Dockerfiles, ranging from unconventional shell commands to forgotten maintainer information. For later testing stages, there are more tools such as Clair, Anchore and the research-proposed DIVA to find runtime stability issues and security vulnerabilities. The tools have certain assumptions about how they are run – Clair for instance is not optimised to be invoked on demand as it downloads CVEs on startup. Instead, it is a long-running process (attached to an equally long-running PostgreSQL server) to which CLI tools connect for doing the actual work. There is for instance claircli, but the more suitable tool for integration into an artefact observation appears to be clair-scanner. For that, a stand-alone variant suitable for CI/CD integration exists with clair-local-scan. The tool pulls a local or remote container image, scans it against the most recent CVE database – if the image based on a supported operating system, outputs the potentially applying CVEs and returns a binary value indicating whether the image is free of issues.
Sometimes, it is not clear which layer an issue comes from, who is responsible for fixing it and which other images may be infected by the issue. While there are some tools to deal with the issue, their results are not satisfactory. The docker images --tree command is deprecated, and the often-recommended chenzj/dfimage container, while nicely concatenating build instructions across the entire history, does not exploit the knowledge about tagged images, nor integrates with external knowledge sources. Enter inheritancetree.sh. This script analyses a Docker image’s build history, reconstructing (as far as possible) the layers and their origin as tagged images, and generates a graph representation. Moreover, the script consults two oracles to provide more information: The Docker Hub API, to get information whether a tagged layer is still up-to-date with regards to the currently published layer for the same tag (and architecture), and a locally running Clair security scanner as described above.
The script automatically picks up installed tools to build up its information sources. A helper script is provided to run the dockerised version of Clair properly. Colouring and labelling is used to provide an easy overview about the state of the image and its ancestry down to the virtual scratch base image. Labels include (outdated) versus (up-to-date) and (vulnerable!) versus (secure).
The inheritancetree.sh script is available from Git. On the applied research side, we continue to observe the chatacteristics and quality aspects of microservice implementation artefacts such as Docker container images to ensure that Swiss companies using such technologies get better insights into what is happening in their cloud application deployments.