Metricbeat docker host process - permission denied

Harm · November 29, 2022, 9:01am

Hi,

I'm trying to use a docker container to monitor my host os processes. It looked quited straightforward by mapping the /proc filesystem to the container and pointing the module hostfs setting to the correct directory, however for some reason the host processes do not appear. After putting metricbeat in debug level, I am seeing the following related message. However it's a debug and I'm not sure if it's really the cause of my issues.

{"log.level":"debug","@timestamp":"2022-11-29T08:34:10.328Z","log.logger":"processes","log.origin":{"file.name":"process/process.go","file.line":146},"message":"Error fetching PID info for 32050, skipping: FillPidMetrics: error getting metadata for pid 32050: error fetching exe from pid 32050: readlink /hostfs/proc/32050/exe: permission denied","service.name":"metricbeat","ecs.version":"1.6.0"}

This does seem to make sense, because when checking the symlink manually, it's also returning a permission denied for this process.

root@1c4160544d1c:/usr/share/metricbeat# ls -l /hostfs/proc/31962/exe
ls: cannot read symbolic link '/hostfs/proc/31962/exe': Permission denied
lrwxrwxrwx 1 root root 0 Nov 29 08:10 /hostfs/proc/31962/exe

I already tried switching the container user to the root-user, but that didn't make any difference. On the host os this link looks like:

root@ip-172-31-44-191:~# ls -l /proc/31962/exe
lrwxrwxrwx 1 root root 0 Nov 29 09:10 /proc/31962/exe -> /usr/sbin/sshd

It looks like a docker issue, after searching around I also tried adding the capability: SYS_PTRACE, however also without luck.

For a complete reference, this is my metricbeat.yml

metricbeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: true
    reload.period: 10s

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
  - add_host_metadata: ~

output.logstash:
  hosts: ["10.0.2.69:5044"]

logging.level: debug
logging.metrics.enabled: false

And system.yml:

- module: system
  period: 1m
  metricsets:
    - process
  hostfs: "/hostfs"

The docker service was created with the following command:

docker service create --name metricbeat --user root --mount type=bind,source=/opt/docker/metricbeat/metricbeat.yml,destination=/usr/share/metricbeat/metricbeat.yml,ro --mount type=bind,source=/opt/docker/metricbeat/system.yml,destination=/usr/share/metricbeat/modules.d/system.yml,ro --mount type=bind,source=/opt/docker/metricbeat/docker.yml,destination=/usr/share/metricbeat/modules.d/docker.yml,ro --mount type=bind,source=/var/run/docker.sock,destination=/var/run/docker.sock,ro --mount type=bind,source=/proc,destination=/hostfs/proc --mount type=bind,source=/sys/fs/cgroup,destination=/hostfs/sys/fs/cgroup,ro --mount type=bind,source=/,destination=/hostfs,ro --network mynet docker.elastic.co/beats/metricbeat:8.5.0

Harm · November 30, 2022, 8:51am

I've also upgraded to the latest docker version, and tried using the SYS_ADMIN capability. Unfortunately without success..

root@ip-172-31-44-191:~# docker version
Client: Docker Engine - Community
 Version:           20.10.21
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        baeda1f
 Built:             Tue Oct 25 18:01:58 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.21
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       3056208
  Built:            Tue Oct 25 17:59:49 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.10
  GitCommit:        770bd0108c32f3fb5c73ae1264f7e503fe7b2661
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

I can find other more or less related issues, but I was not able to fix the issue. Any suggestions?

github.com/moby/moby

Allow access to /proc/PID/exe by default

opened 01:33PM - 18 Mar 20 UTC

interone-ms

**Description** Access to `/proc/PID/exe` for processes running in the contai…ner as another user is denied by docker unless running the container with `--cap-add=SYS_PTRACE`. This breaks OS-provided init scripts that rely on `start-stop-daemon`'s `--exec` feature, as it works by looping through processes and checks if the path supplied with the `--exec` flag matches the path that `/proc/PID/exe` points to. As mere access to `/proc/PID/exe` should not introduce any security issues, it should be whitelisted by default. Otherwise, if whitelisting is not desired, it should *at least* be moved to its own `CAP_` flag (in the [manpage](http://man7.org/linux/man-pages/man7/capabilities.7.html), the section about `CAP_SYS_PTRACE` doesn't mention `/proc/PID/exe`). This is a duplicate of #7147 which got closed as a dupe of #6800 and #11049, which are both closed without providing *ANY* workaround for people needing this during the build phase (where `--cap-add` is not supported at all, see #1916). The proper fix to this bug is to whitelist `readlink` calls to `/proc/PID/exe` no matter the user inside the container or the user of the process, while an (borderline acceptable) workaround would be implementing capabilities during build (#1916). **Steps to reproduce the issue:** 1. Run a container with Ubuntu 18.04: `docker run --rm -it ubuntu:18.04 bash` 2. Run everything from now inside the container 3. Install tomcat8: `apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -yf tomcat8 openjdk-8-jdk-headless openjdk-11-jre-headless-` 4. Install the prerequisites (mainly permission stuff, so that Tomcat can start): ``` usermod --home /var/lib/tomcat8 $TOMCAT8_USER > /dev/null 2>&1 || true mkdir -p /tmp/tomcat8-tomcat8-tmp chown -h tomcat8 /tmp/tomcat8-tomcat8-tmp install -o tomcat8 -g adm -m 644 /dev/null /var/run/tomcat8.pid su tomcat8 -s /bin/bash -c "install -m 644 /dev/null /var/log/tomcat8/catalina.out" ``` 5. Run tomcat8 as `tomcat8` user: ``` su tomcat8 -s /bin/bash -c 'set -a; JAVA_HOME="/usr/lib/jvm/java-8-openjdk-amd64"; \ source "/etc/default/tomcat8"; CATALINA_HOME="/usr/share/tomcat8"; \ CATALINA_BASE="/var/lib/tomcat8"; \ JAVA_OPTS="-Djava.awt.headless=true -XX:+UseConcMarkSweepGC"; \ CATALINA_PID="/var/run/tomcat8.pid"; \ CATALINA_TMPDIR="/tmp/tomcat8-tomcat8-tmp"; \ LANG=""; JSSE_HOME=""; \ cd "/var/lib/tomcat8"; \ "/usr/share/tomcat8/bin/catalina.sh" start' ``` 6. Try to read the `/proc/PID/exe` path as root: `ls -lahn /proc/$(cat /var/run/tomcat8.pid)/exe` 7. Try to read the `/proc/PID/exe` path as tomcat8: `su tomcat8 -s /bin/bash -c "ls -lahn /proc/"$(cat /var/run/tomcat8.pid)"/exe"` **Describe the results you received:** Step 6 (reading the path as root) fails, step 7 succeeds: ``` root@3651a04ac34b:/# ls -lahn /proc/$(cat /var/run/tomcat8.pid)/exe ls: cannot read symbolic link '/proc/3955/exe': Permission denied lrwxrwxrwx 1 102 102 0 Mar 18 13:26 /proc/3955/exe root@3651a04ac34b:/# su tomcat8 -s /bin/bash -c "ls -lahn /proc/"$(cat /var/run/tomcat8.pid)"/exe" lrwxrwxrwx 1 102 102 0 Mar 18 13:26 /proc/3955/exe -> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java ``` **Describe the results you expected:** Both steps 6 and 7 show the proper path: ``` lrwxrwxrwx 1 102 102 0 Mar 18 13:26 /proc/3955/exe -> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java ``` **Additional information you deem important (e.g. issue happens only occasionally):** **Output of `docker version`:** ``` Client: Docker Engine - Community Version: 19.03.5 API version: 1.40 Go version: go1.12.12 Git commit: 633a0ea Built: Wed Nov 13 07:22:34 2019 OS/Arch: darwin/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 19.03.5 API version: 1.40 (minimum version 1.12) Go version: go1.12.12 Git commit: 633a0ea Built: Wed Nov 13 07:29:19 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.2.10 GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339 runc: Version: 1.0.0-rc8+dev GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 docker-init: Version: 0.18.0 GitCommit: fec3683 ``` **Output of `docker info`:** ``` Client: Debug Mode: false Server: Containers: 12 Running: 0 Paused: 0 Stopped: 12 Images: 122 Server Version: 19.03.5 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 4.19.76-linuxkit Operating System: Docker Desktop OSType: linux Architecture: x86_64 CPUs: 4 Total Memory: 1.943GiB Name: docker-desktop ID: xxxxx Docker Root Dir: /var/lib/docker Debug Mode: true File Descriptors: 35 Goroutines: 52 System Time: 2020-03-18T13:29:20.5840226Z EventsListeners: 3 HTTP Proxy: gateway.docker.internal:3128 HTTPS Proxy: gateway.docker.internal:3129 Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Product License: Community Engine ``` **Additional environment details (AWS, VirtualBox, physical, etc.):** Docker on Mac

github.com/moby/moby

Unable to read symlink inside the container

opened 08:48PM - 27 Sep 22 UTC

psaini79

status/0-triage kind/bug

### Description Unable to read symlink inside the containers. We are using pwdx… command to read the cwd directory of the process and it is mapped through a symlink inside the /proc/ But when we execute pwdx, we are getting permission denied error. We tried giving sys_admin capability and it seems to be working. However, we can't give such a broad capability. Do we know what capability is required to read the softlinks inside the containers: ### Reproduce 1. Docker Run command ``` docker create -t -i \ --hostname test1 \ --dns-search="example.info" \ --privileged=false \ --cap-add=SYS_NICE \ --cap-add=SYS_RESOURCE \ --cap-add=NET_ADMIN \ --restart=always \ --tmpfs=/run \ --ulimit rtprio=99 \ --name test1 \ oraclelinux:7 ``` 2. Exec to the container and execute the steps mentioned below: ``` docker exec -i -t test1 /bin/bash useradd test1 su - test1 vi /home/test1/test.sh #!/bin/bash while true; do echo "Hi" sleep 30 done ``` 3. Start the script ``` nohup ./test.sh & ``` 4. Grab the process as test user ``` [test1@test1 ~]$ ps -ef | grep test.sh test1 56 35 0 03:43 pts/1 00:00:00 /bin/bash ./test.sh [test1@test1 ~]$ pwdx 56 56: /home/test1 [test1@test1 ~]$ ls -ltr /proc/56/cwd lrwxrwxrwx 1 test1 test1 0 Sep 28 03:44 /proc/56/cwd -> /home/test1 [test1@test1 ~]$ ``` 5. Exit from the root user and test it as a root user ``` [root@test1 ~]# pwdx 56 56: Permission denied [root@test1 ~]# ls -ltr /proc/56/cwd ls: cannot read symbolic link /proc/56/cwd: Permission denied. <<< Permission denied error lrwxrwxrwx 1 test1 test1 0 Sep 28 03:44 /proc/56/cwd [root@test1 ~]# ``` ### Expected behavior `docker stop test1; docker rm test1` ### docker version ```bash [root@docker01 bldc]# docker version Client: Docker Engine - Community Version: 19.03.11-ol API version: 1.40 Go version: go1.16.2 Git commit: 9bb540d Built: Fri Jul 23 01:33:55 2021 OS/Arch: linux/amd64 Experimental: false Server: Docker Engine - Community Engine: Version: 19.03.11-ol API version: 1.40 (minimum version 1.12) Go version: go1.16.2 Git commit: 9bb540d Built: Fri Jul 23 01:32:08 2021 OS/Arch: linux/amd64 Experimental: false Default Registry: docker.io containerd: Version: v1.4.8 GitCommit: 7eba5930496d9bbe375fdf71603e610ad737d2b2 runc: Version: 1.0.2 GitCommit: 2856f01 docker-init: Version: 0.18.0 GitCommit: fec3683 [root@docker01 bldc]# ``` ### docker info ```bash [root@docker01 bldc]# docker info Client: Debug Mode: false Server: Containers: 3 Running: 2 Paused: 0 Stopped: 1 Images: 38 Server Version: 19.03.11-ol Storage Driver: overlay2 Backing Filesystem: xfs Supports d_type: true Native Overlay Diff: false Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 7eba5930496d9bbe375fdf71603e610ad737d2b2 runc version: 2856f01 init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 5.4.17-2102.201.3.el7uek.x86_64 Operating System: Oracle Linux Server 7.9 OSType: linux Architecture: x86_64 CPUs: 6 Total Memory: 31.08GiB Name: docker01 ID: 5EUB:42QG:25UN:ASIE:Q7ML:JJSB:736R:5HQS:33LT:IY46:XZK2:RRAP Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false Registries: ``` ### Additional Info _No response_

system · December 28, 2022, 10:52am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.