Hi,
I'm trying to use a docker container to monitor my host os processes. It looked quited straightforward by mapping the /proc filesystem to the container and pointing the module hostfs setting to the correct directory, however for some reason the host processes do not appear. After putting metricbeat in debug level, I am seeing the following related message. However it's a debug and I'm not sure if it's really the cause of my issues.
{"log.level":"debug","@timestamp":"2022-11-29T08:34:10.328Z","log.logger":"processes","log.origin":{"file.name":"process/process.go","file.line":146},"message":"Error fetching PID info for 32050, skipping: FillPidMetrics: error getting metadata for pid 32050: error fetching exe from pid 32050: readlink /hostfs/proc/32050/exe: permission denied","service.name":"metricbeat","ecs.version":"1.6.0"}
This does seem to make sense, because when checking the symlink manually, it's also returning a permission denied for this process.
root@1c4160544d1c:/usr/share/metricbeat# ls -l /hostfs/proc/31962/exe
ls: cannot read symbolic link '/hostfs/proc/31962/exe': Permission denied
lrwxrwxrwx 1 root root 0 Nov 29 08:10 /hostfs/proc/31962/exe
I already tried switching the container user to the root-user, but that didn't make any difference. On the host os this link looks like:
root@ip-172-31-44-191:~# ls -l /proc/31962/exe
lrwxrwxrwx 1 root root 0 Nov 29 09:10 /proc/31962/exe -> /usr/sbin/sshd
It looks like a docker issue, after searching around I also tried adding the capability: SYS_PTRACE, however also without luck.
For a complete reference, this is my metricbeat.yml
metricbeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: true
reload.period: 10s
metricbeat.autodiscover:
providers:
- type: docker
hints.enabled: true
processors:
- add_host_metadata: ~
output.logstash:
hosts: ["10.0.2.69:5044"]
logging.level: debug
logging.metrics.enabled: false
And system.yml:
- module: system
period: 1m
metricsets:
- process
hostfs: "/hostfs"
The docker service was created with the following command:
docker service create --name metricbeat --user root --mount type=bind,source=/opt/docker/metricbeat/metricbeat.yml,destination=/usr/share/metricbeat/metricbeat.yml,ro --mount type=bind,source=/opt/docker/metricbeat/system.yml,destination=/usr/share/metricbeat/modules.d/system.yml,ro --mount type=bind,source=/opt/docker/metricbeat/docker.yml,destination=/usr/share/metricbeat/modules.d/docker.yml,ro --mount type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock,ro --mount type=bind,source=/proc,destination=/hostfs/proc --mount type=bind,source=/sys/fs/cgroup,destination=/hostfs/sys/fs/cgroup,ro --mount type=bind,source=/,destination=/hostfs,ro --network mynet docker.elastic.co/beats/metricbeat:8.5.0