For the last two years we have been using Docker containerization solutions for the needs of development, CI and production environments. All current products for the customers are being developed to be run in Docker.
We have a very positive experience about that solution. However, Docker changes a lot, from version to version it introduces new amazing stuff like Compose, Swarm, additional security features, which, sometimes, lead to errors in containers that was considered as stable.
Thus, with great amazement we have discovered that a container which was developed and tested 3 months ago is no longer functional in a new Docker version. The container is not a typical thing because it uses strace to analyze a process behaviour to diagnose deviant behaviour of it.
Today our developers started an integration process of that container in an application and found that it doesn’t work anymore and the reason is connected to ptrace which is reported by strace like:
strace: attach: ptrace(PTRACE_ATTACH, ...): Operation not permitted Could not attach to process. If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try again as the root user. For more details, see /etc/sysctl.d/10-ptrace.conf
You probably wonder, the case is quite rare — strace in Docker… Er, but there are some similar questions in the internet which are more or less address the same problem. So, what’s the reason and how to tackle with it? There are several things.
Seccomp Mechanism
In distros with new Linux operating system kernel there is modern kernel mechanism — seccomp, which
enables Docker blocking of system calls inside a container. The Docker documentation about this feature is located at
official Docker documentation page. As one can discover there,
ptrace is blocked. Thin seccomp configuration can be implemented by overriding the default configuration file by using
docker run command line argument –security-opt seccomp=/path/to/file.json. Our container is launched in a protected
environment, so we disabled it completely by setting --security-opt seccomp=unconfined.
Yama Mechanism
Strace shown us an clue about other possible ptrace limitation — procfs pseudofile /proc/sys/kernel/yama/ptrace_scope,
which introduces additional ptrace limitation capabilities. Let’s take a look at the kernel
docs and discover that the file content is presented by
digits 0-3 which in case define ptrace capabilities:
The sysctl settings (writable only with CAP_SYS_PTRACE) are:
0 - classic ptrace permissions: a process can PTRACE_ATTACH to any other
process running under the same uid, as long as it is dumpable (i.e.
did not transition uids, start privileged, or have called
prctl(PR_SET_DUMPABLE...) already). Similarly, PTRACE_TRACEME is
unchanged.
1 - restricted ptrace: a process must have a predefined relationship
with the inferior it wants to call PTRACE_ATTACH on. By default,
this relationship is that of only its descendants when the above
classic criteria is also met. To change the relationship, an
inferior can call prctl(PR_SET_PTRACER, debugger, ...) to declare
an allowed debugger PID to call PTRACE_ATTACH on the inferior.
Using PTRACE_TRACEME is unchanged.
2 - admin-only attach: only processes with CAP_SYS_PTRACE may use ptrace
with PTRACE_ATTACH, or through children calling PTRACE_TRACEME.
3 - no attach: no processes may use ptrace with PTRACE_ATTACH nor via
PTRACE_TRACEME. Once set, this sysctl value cannot be changed.
By executing following command:
docker exec -it <container> cat /proc/sys/kernel/yama/ptrace_scope
We found that in our case file has value of “1” set, and since our processes were executed with the same UID but not in the parent-child hierarchy, Yama blocked PTRACE_ATTACH.
We tried to put “0” value into /proc/sys/kernel/yama/ptrace_scope but failed because we got “/proc is read only”
message.
Privileged Execution Mode
We overcame that “read-only procfs” limitation by running our container in “privileged” execution mode setting
“–privileged” docker run flag. We also put “0” to /proc/sys/kernel/yama/ptrace_scope by modification of application
bootstrap code.
echo 0 > /proc/sys/kernel/yama/ptrace_scope
All above means allowed us to solve the problem. The example solution can be found in our web ssh proxy container.