Recently a Docker container refused to start for me without any hint about why in the OS or Docker container’s logs. It turns out that SELinux jumped in and blocked access without logging its interference. Containers which were running in the past would not start after the update to RHEL 7.5 without any error message in the logs. This is a closer look into what causes this issue with Docker and SElinux.
The Problem
Docker containers contain the very minimum to run an application or service. The containers are isolated using namespaces and limited using cgroups (Docker: The underlying technology). SELinux on RedHat based distributions enforces another layer of security by denying access to certain resources.
In many cases a container does not run any components requiring cgroup configuration access but in some cases components might require it. The most common example is running a container which includes systemd. Systemd heavily uses cgroups. If a container contains systemd, it requires access which worked out of the box with RHEL up to 7.4.
Systemd shows the denied access to the cgroup configuration with an error message like this (path reduced in this example).
Failed to create cgroup /.../systemd-journald.service: Permission denied
But not all container applications trying to access the cgroup configuration are logging the configuration in this way. Interestingly, on the host system this denied access to the cgroups is not logged. At the time of writing, no entry could be found in the logs on the host system indicating SELinux interference with the cgroup access.
If the container does not contain any log entries like this, debugging this issue is a challenge. Some applications were found to be not logging such a situation, making it nearly impossible to find the cause. A good hint might be if the documentation for the application states that running the container in privileged mode (–privileged) is suggested.
The Cause
By default, the SELinux policy will deny access from within a container to the cgroup configuration. RHEL releases before 7.5 contained a bug which did not deny access from within a container to the cgroup configuration. With the release of RHEL 7.5 this bug was fixed. This change/fix in the SELinux policy caused Docker containers which “worked” previously to fail. See RedHat Knowledgebase article Container running systemd fails to run after upgrade to Red Hat Enterprise Linux 7.5.
As with many other packages, Docker installs a number of SELinux policies, including SELinux booleans. Those booleans are meant to change the policy behaviour based on common use cases. Access to the cgroup configuration is one of those use cases, so a boolean switch is provided to control this access. The SELinux policy installed via “container-selinux” is maintained on Github.
It is unclear at the moment why this issue is not logged on the Docker host at all. The fact that there is no log entry in the host means that this can be very annoying and time consuming problem to fix.
The Solution
The solution is as simple as flipping a switch. The related seboolean to control the access to the cgroup configuration needs to be turned “on” to allow access to the cgroup configuration. As the boolean switch did not work as intended before, it is probably not set to on before the 7.5 update causing the described issue.
$ setsebool -P container_manage_cgroup on
After setting this SELinux boolean, any Docker containers which failed due to this issue should start again.
Read more of my posts on my blog at https://blog.tinned-software.net/.
