Metricbeat docker host process - permission denied


I'm trying to use a docker container to monitor my host os processes. It looked quited straightforward by mapping the /proc filesystem to the container and pointing the module hostfs setting to the correct directory, however for some reason the host processes do not appear. After putting metricbeat in debug level, I am seeing the following related message. However it's a debug and I'm not sure if it's really the cause of my issues.

{"log.level":"debug","@timestamp":"2022-11-29T08:34:10.328Z","log.logger":"processes","log.origin":{"":"process/process.go","file.line":146},"message":"Error fetching PID info for 32050, skipping: FillPidMetrics: error getting metadata for pid 32050: error fetching exe from pid 32050: readlink /hostfs/proc/32050/exe: permission denied","":"metricbeat","ecs.version":"1.6.0"}

This does seem to make sense, because when checking the symlink manually, it's also returning a permission denied for this process.

root@1c4160544d1c:/usr/share/metricbeat# ls -l /hostfs/proc/31962/exe
ls: cannot read symbolic link '/hostfs/proc/31962/exe': Permission denied
lrwxrwxrwx 1 root root 0 Nov 29 08:10 /hostfs/proc/31962/exe

I already tried switching the container user to the root-user, but that didn't make any difference. On the host os this link looks like:

root@ip-172-31-44-191:~# ls -l /proc/31962/exe
lrwxrwxrwx 1 root root 0 Nov 29 09:10 /proc/31962/exe -> /usr/sbin/sshd

It looks like a docker issue, after searching around I also tried adding the capability: SYS_PTRACE, however also without luck.

For a complete reference, this is my metricbeat.yml

    path: ${path.config}/modules.d/*.yml
    reload.enabled: true
    reload.period: 10s

    - type: docker
      hints.enabled: true

  - add_host_metadata: ~

  hosts: [""]

logging.level: debug
logging.metrics.enabled: false

And system.yml:

- module: system
  period: 1m
    - process
  hostfs: "/hostfs"

The docker service was created with the following command:

docker service create --name metricbeat --user root --mount type=bind,source=/opt/docker/metricbeat/metricbeat.yml,destination=/usr/share/metricbeat/metricbeat.yml,ro --mount type=bind,source=/opt/docker/metricbeat/system.yml,destination=/usr/share/metricbeat/modules.d/system.yml,ro --mount type=bind,source=/opt/docker/metricbeat/docker.yml,destination=/usr/share/metricbeat/modules.d/docker.yml,ro --mount type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock,ro --mount type=bind,source=/proc,destination=/hostfs/proc --mount type=bind,source=/sys/fs/cgroup,destination=/hostfs/sys/fs/cgroup,ro --mount type=bind,source=/,destination=/hostfs,ro --network mynet

I've also upgraded to the latest docker version, and tried using the SYS_ADMIN capability. Unfortunately without success..

root@ip-172-31-44-191:~# docker version
Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:58 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:49 2022
  OS/Arch:          linux/amd64
  Experimental:     false
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
  Version:          0.19.0
  GitCommit:        de40ad0

I can find other more or less related issues, but I was not able to fix the issue. Any suggestions?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.