# Metricbeat docker host process - permission denied

**URL:** <https://discuss.elastic.co/t/metricbeat-docker-host-process-permission-denied/320015>\
**Category:** Beats\
**Tags:** docker, metricbeat\
**Created:** [November 29, 2022, 9:01am UTC](https://discuss.elastic.co/t/metricbeat-docker-host-process-permission-denied/320015 "2022-11-29T09:01:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Harm](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@Harm](https://discuss.elastic.co/u/Harm)\
**Post date:** [November 29, 2022, 9:01am UTC](https://discuss.elastic.co/t/metricbeat-docker-host-process-permission-denied/320015/1 "2022-11-29T09:01:31Z")

</div>

Hi,

I'm trying to use a docker container to monitor my host os processes. It looked quited straightforward by mapping the /proc filesystem to the container and pointing the module hostfs setting to the correct directory, however for some reason the host processes do not appear. After putting metricbeat in debug level, I am seeing the following related message. However it's a debug and I'm not sure if it's really the cause of my issues.

` {"log.level":"debug","@timestamp":"2022-11-29T08:34:10.328Z","log.logger":"processes","log.origin":{"file.name":"process/process.go","file.line":146},"message":"Error fetching PID info for 32050, skipping: FillPidMetrics: error getting metadata for pid 32050: error fetching exe from pid 32050: readlink /hostfs/proc/32050/exe: permission denied","service.name":"metricbeat","ecs.version":"1.6.0"}`

This does seem to make sense, because when checking the symlink manually, it's also returning a permission denied for this process.

```auto
root@1c4160544d1c:/usr/share/metricbeat# ls -l /hostfs/proc/31962/exe
ls: cannot read symbolic link '/hostfs/proc/31962/exe': Permission denied
lrwxrwxrwx 1 root root 0 Nov 29 08:10 /hostfs/proc/31962/exe

```

I already tried switching the container user to the root-user, but that didn't make any difference. On the host os this link looks like:

```auto
root@ip-172-31-44-191:~# ls -l /proc/31962/exe
lrwxrwxrwx 1 root root 0 Nov 29 09:10 /proc/31962/exe -> /usr/sbin/sshd

```

It looks like a docker issue, after searching around I also tried adding the capability: SYS\_PTRACE, however also without luck.

For a complete reference, this is my metricbeat.yml

```auto
metricbeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true
    reload.period: 10s

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
  - add_host_metadata: ~

output.logstash:
  hosts: ["10.0.2.69:5044"]

logging.level: debug
logging.metrics.enabled: false

```

And system.yml:

```auto
- module: system
  period: 1m
  metricsets:
    - process
  hostfs: "/hostfs"

```

The docker service was created with the following command:

`docker service create --name metricbeat --user root --mount type=bind,source=/opt/docker/metricbeat/metricbeat.yml,destination=/usr/share/metricbeat/metricbeat.yml,ro --mount type=bind,source=/opt/docker/metricbeat/system.yml,destination=/usr/share/metricbeat/modules.d/system.yml,ro --mount type=bind,source=/opt/docker/metricbeat/docker.yml,destination=/usr/share/metricbeat/modules.d/docker.yml,ro --mount type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock,ro --mount type=bind,source=/proc,destination=/hostfs/proc --mount type=bind,source=/sys/fs/cgroup,destination=/hostfs/sys/fs/cgroup,ro --mount type=bind,source=/,destination=/hostfs,ro --network mynet docker.elastic.co/beats/metricbeat:8.5.0`

---

<div class="post-metadata">

**Author:** ![Harm](https://avatars.discourse-cdn.com/v4/letter/h/b4bc9f/32.png) [@Harm](https://discuss.elastic.co/u/Harm)\
**Post date:** [November 30, 2022, 8:51am UTC](https://discuss.elastic.co/t/metricbeat-docker-host-process-permission-denied/320015/2 "2022-11-30T08:51:30Z")

</div>

I've also upgraded to the latest docker version, and tried using the SYS\_ADMIN capability. Unfortunately without success..

```auto
root@ip-172-31-44-191:~# docker version
Client: Docker Engine - Community
 Version: 20.10.21
 API version: 1.41
 Go version: go1.18.7
 Git commit: baeda1f
 Built: Tue Oct 25 18:01:58 2022
 OS/Arch: linux/amd64
 Context: default
 Experimental: true

Server: Docker Engine - Community
 Engine:
  Version: 20.10.21
  API version: 1.41 (minimum version 1.12)
  Go version: go1.18.7
  Git commit: 3056208
  Built: Tue Oct 25 17:59:49 2022
  OS/Arch: linux/amd64
  Experimental: false
 containerd:
  Version: 1.6.10
  GitCommit: 770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version: 1.1.4
  GitCommit: v1.1.4-0-g5fd4c4d
 docker-init:
  Version: 0.19.0
  GitCommit: de40ad0

```

I can find other more or less related issues, but I was not able to fix the issue. Any suggestions?

> <https://github.com/moby/moby/issues/40713>
>
> \*\*Description\*\*
> 
> Access to \`/proc/PID/exe\` for processes running in the contai…ner as another user is denied by docker unless running the container with \`--cap-add=SYS\_PTRACE\`. This breaks OS-provided init scripts that rely on \`start-stop-daemon\`'s \`--exec\` feature, as it works by looping through processes and checks if the path supplied with the \`--exec\` flag matches the path that \`/proc/PID/exe\` points to.
> 
> As mere access to \`/proc/PID/exe\` should not introduce any security issues, it should be whitelisted by default. Otherwise, if whitelisting is not desired, it should \*at least\* be moved to its own \`CAP\_\` flag (in the \[manpage\](http://man7.org/linux/man-pages/man7/capabilities.7.html), the section about \`CAP\_SYS\_PTRACE\` doesn't mention \`/proc/PID/exe\`).
> 
> This is a duplicate of #7147 which got closed as a dupe of #6800 and #11049, which are both closed without providing \*ANY\* workaround for people needing this during the build phase (where \`--cap-add\` is not supported at all, see #1916).
> 
> The proper fix to this bug is to whitelist \`readlink\` calls to \`/proc/PID/exe\` no matter the user inside the container or the user of the process, while an (borderline acceptable) workaround would be implementing capabilities during build (#1916).
> 
> \*\*Steps to reproduce the issue:\*\*
> 1. Run a container with Ubuntu 18.04: \`docker run --rm -it ubuntu:18.04 bash\`
> 2. Run everything from now inside the container
> 3. Install tomcat8: \`apt-get update && DEBIAN\_FRONTEND=noninteractive apt-get install -yf tomcat8 openjdk-8-jdk-headless openjdk-11-jre-headless-\`
> 4. Install the prerequisites (mainly permission stuff, so that Tomcat can start):
> \`\`\`
> usermod --home /var/lib/tomcat8 $TOMCAT8\_USER \> /dev/null 2\>&1 || true
> mkdir -p /tmp/tomcat8-tomcat8-tmp
> chown -h tomcat8 /tmp/tomcat8-tomcat8-tmp
> install -o tomcat8 -g adm -m 644 /dev/null /var/run/tomcat8.pid
> su tomcat8 -s /bin/bash -c "install -m 644 /dev/null /var/log/tomcat8/catalina.out"
> \`\`\` 
> 5. Run tomcat8 as \`tomcat8\` user:
> \`\`\`
> su tomcat8 -s /bin/bash -c 'set -a; JAVA\_HOME="/usr/lib/jvm/java-8-openjdk-amd64"; \\
> source "/etc/default/tomcat8"; CATALINA\_HOME="/usr/share/tomcat8"; \\
> CATALINA\_BASE="/var/lib/tomcat8"; \\
> JAVA\_OPTS="-Djava.awt.headless=true -XX:+UseConcMarkSweepGC"; \\
> CATALINA\_PID="/var/run/tomcat8.pid"; \\
> CATALINA\_TMPDIR="/tmp/tomcat8-tomcat8-tmp"; \\
> LANG=""; JSSE\_HOME=""; \\
> cd "/var/lib/tomcat8"; \\
> "/usr/share/tomcat8/bin/catalina.sh" start'
> \`\`\`
> 6. Try to read the \`/proc/PID/exe\` path as root: \`ls -lahn /proc/$(cat /var/run/tomcat8.pid)/exe\`
> 7. Try to read the \`/proc/PID/exe\` path as tomcat8: \`su tomcat8 -s /bin/bash -c "ls -lahn /proc/"$(cat /var/run/tomcat8.pid)"/exe"\` 
> 
> \*\*Describe the results you received:\*\*
> 
> Step 6 (reading the path as root) fails, step 7 succeeds:
> 
> \`\`\`
> root@3651a04ac34b:/# ls -lahn /proc/$(cat /var/run/tomcat8.pid)/exe
> ls: cannot read symbolic link '/proc/3955/exe': Permission denied
> lrwxrwxrwx 1 102 102 0 Mar 18 13:26 /proc/3955/exe
> root@3651a04ac34b:/# su tomcat8 -s /bin/bash -c "ls -lahn /proc/"$(cat /var/run/tomcat8.pid)"/exe"
> lrwxrwxrwx 1 102 102 0 Mar 18 13:26 /proc/3955/exe -\> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
> \`\`\`
> 
> \*\*Describe the results you expected:\*\*
> Both steps 6 and 7 show the proper path:
> \`\`\`
> lrwxrwxrwx 1 102 102 0 Mar 18 13:26 /proc/3955/exe -\> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
> \`\`\`
> 
> \*\*Additional information you deem important (e.g. issue happens only occasionally):\*\*
> 
> \*\*Output of \`docker version\`:\*\*
> 
> \`\`\`
> Client: Docker Engine - Community
> Version: 19.03.5
> API version: 1.40
> Go version: go1.12.12
> Git commit: 633a0ea
> Built: Wed Nov 13 07:22:34 2019
> OS/Arch: darwin/amd64
> Experimental: false
> 
> Server: Docker Engine - Community
> Engine:
> Version: 19.03.5
> API version: 1.40 (minimum version 1.12)
> Go version: go1.12.12
> Git commit: 633a0ea
> Built: Wed Nov 13 07:29:19 2019
> OS/Arch: linux/amd64
> Experimental: false
> containerd:
> Version: v1.2.10
> GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
> runc:
> Version: 1.0.0-rc8+dev
> GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
> docker-init:
> Version: 0.18.0
> GitCommit: fec3683
> \`\`\`
> 
> \*\*Output of \`docker info\`:\*\*
> 
> \`\`\`
> Client:
> Debug Mode: false
> 
> Server:
> Containers: 12
> Running: 0
> Paused: 0
> Stopped: 12
> Images: 122
> Server Version: 19.03.5
> Storage Driver: overlay2
> Backing Filesystem: extfs
> Supports d\_type: true
> Native Overlay Diff: true
> Logging Driver: json-file
> Cgroup Driver: cgroupfs
> Plugins:
> Volume: local
> Network: bridge host ipvlan macvlan null overlay
> Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
> Swarm: inactive
> Runtimes: runc
> Default Runtime: runc
> Init Binary: docker-init
> containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
> runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
> init version: fec3683
> Security Options:
> seccomp
> Profile: default
> Kernel Version: 4.19.76-linuxkit
> Operating System: Docker Desktop
> OSType: linux
> Architecture: x86\_64
> CPUs: 4
> Total Memory: 1.943GiB
> Name: docker-desktop
> ID: xxxxx
> Docker Root Dir: /var/lib/docker
> Debug Mode: true
> File Descriptors: 35
> Goroutines: 52
> System Time: 2020-03-18T13:29:20.5840226Z
> EventsListeners: 3
> HTTP Proxy: gateway.docker.internal:3128
> HTTPS Proxy: gateway.docker.internal:3129
> Registry: https://index.docker.io/v1/
> Labels:
> Experimental: false
> Insecure Registries:
> 127.0.0.0/8
> Live Restore Enabled: false
> Product License: Community Engine
> \`\`\`
> 
> \*\*Additional environment details (AWS, VirtualBox, physical, etc.):\*\*
> Docker on Mac

> <https://github.com/moby/moby/issues/44209>
>
> \### Description
> 
> Unable to read symlink inside the containers. We are using pwdx… command to read the cwd directory of the process and it is mapped through a symlink inside the /proc/ But when we execute pwdx, we are getting permission denied error. We tried giving sys\_admin capability and it seems to be working. However, we can't give such a broad capability. Do we know what capability is required to read the softlinks inside the containers: 
> 
> \### Reproduce
> 
> 1. Docker Run command
> \`\`\`
> docker create -t -i \\
> --hostname test1 \\
> --dns-search="example.info" \\
> --privileged=false \\
> --cap-add=SYS\_NICE \\
> --cap-add=SYS\_RESOURCE \\
> --cap-add=NET\_ADMIN \\
> --restart=always \\
> --tmpfs=/run \\
> --ulimit rtprio=99 \\
> --name test1 \\
> oraclelinux:7
> \`\`\` 
> 2. Exec to the container and execute the steps mentioned below:
> \`\`\`
> docker exec -i -t test1 /bin/bash
> useradd test1
> su - test1
> vi /home/test1/test.sh
> 
> \#!/bin/bash
> 
> while true; do
> echo "Hi"
> 
> sleep 30
> 
> done
> \`\`\`
> 3. Start the script
> \`\`\`
> nohup ./test.sh &
> \`\`\`
> 4. Grab the process as test user
> \`\`\`
> \[test1@test1 ~\]$ ps -ef | grep test.sh 
> test1 56 35 0 03:43 pts/1 00:00:00 /bin/bash ./test.sh
> 
> \[test1@test1 ~\]$ pwdx 56
> 56: /home/test1
> 
> \[test1@test1 ~\]$ ls -ltr /proc/56/cwd
> lrwxrwxrwx 1 test1 test1 0 Sep 28 03:44 /proc/56/cwd -\> /home/test1
> \[test1@test1 ~\]$ 
> 
> \`\`\`
> 5. Exit from the root user and test it as a root user
> \`\`\`
> \[root@test1 ~\]# pwdx 56
> 56: Permission denied
> \[root@test1 ~\]# ls -ltr /proc/56/cwd
> ls: cannot read symbolic link /proc/56/cwd: Permission denied. \<\<\< Permission denied error
> lrwxrwxrwx 1 test1 test1 0 Sep 28 03:44 /proc/56/cwd
> \[root@test1 ~\]# 
> 
> \`\`\`
> 
> \### Expected behavior
> 
> \`docker stop test1; docker rm test1\`
> 
> \### docker version
> 
> \`\`\`bash
> \[root@docker01 bldc\]# docker version
> Client: Docker Engine - Community
> Version: 19.03.11-ol
> API version: 1.40
> Go version: go1.16.2
> Git commit: 9bb540d
> Built: Fri Jul 23 01:33:55 2021
> OS/Arch: linux/amd64
> Experimental: false
> 
> Server: Docker Engine - Community
> Engine:
> Version: 19.03.11-ol
> API version: 1.40 (minimum version 1.12)
> Go version: go1.16.2
> Git commit: 9bb540d
> Built: Fri Jul 23 01:32:08 2021
> OS/Arch: linux/amd64
> Experimental: false
> Default Registry: docker.io
> containerd:
> Version: v1.4.8
> GitCommit: 7eba5930496d9bbe375fdf71603e610ad737d2b2
> runc:
> Version: 1.0.2
> GitCommit: 2856f01
> docker-init:
> Version: 0.18.0
> GitCommit: fec3683
> \[root@docker01 bldc\]#
> \`\`\`
> 
> 
> \### docker info
> 
> \`\`\`bash
> \[root@docker01 bldc\]# docker info
> Client:
> Debug Mode: false
> 
> Server:
> Containers: 3
> Running: 2
> Paused: 0
> Stopped: 1
> Images: 38
> Server Version: 19.03.11-ol
> Storage Driver: overlay2
> Backing Filesystem: xfs
> Supports d\_type: true
> Native Overlay Diff: false
> Logging Driver: json-file
> Cgroup Driver: cgroupfs
> Plugins:
> Volume: local
> Network: bridge host ipvlan macvlan null overlay
> Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
> Swarm: inactive
> Runtimes: runc
> Default Runtime: runc
> Init Binary: docker-init
> containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2
> runc version: 2856f01
> init version: fec3683
> Security Options:
> seccomp
> Profile: default
> Kernel Version: 5.4.17-2102.201.3.el7uek.x86\_64
> Operating System: Oracle Linux Server 7.9
> OSType: linux
> Architecture: x86\_64
> CPUs: 6
> Total Memory: 31.08GiB
> Name: docker01
> ID: 5EUB:42QG:25UN:ASIE:Q7ML:JJSB:736R:5HQS:33LT:IY46:XZK2:RRAP
> Docker Root Dir: /var/lib/docker
> Debug Mode: false
> Registry: https://index.docker.io/v1/
> Labels:
> Experimental: false
> Insecure Registries:
> 127.0.0.0/8
> Live Restore Enabled: false
> 
> Registries:
> \`\`\`
> 
> 
> \### Additional Info
> 
> \_No response\_

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 28, 2022, 10:52am UTC](https://discuss.elastic.co/t/metricbeat-docker-host-process-permission-denied/320015/3 "2022-12-28T10:52:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.