OS: CentOS 7, kernel 3.10
Docker version : 1.12.6
Elastic version : 5.6.2
Metricbeat version : 5.6.2
I'm trying to run metricbeat as a docker container to monitor other docker containers.
As I don't have the same docker group id across my environments i can't
usermod -g metricbeat so that metricbeat can communicate through the docker socket.
I chose to run the process as root inside the container.
The container is privileged, and has rw access to the /var/run/docker.sock mounted as a volume (tested with netcat)
The error I get is :
... docker.go:132: ERR An error occurred while getting docker stats: read unix @->/var/run/docker.sock: i/o timeout
It seems that all data is pushed to ES correctly despite this error.
After testing, it seems that these errors occur despite sending complete data to elasticsearch, and using debug doesn't change anything.
updating first post.
Could you share your metricbeat.yml and the docker command you are using? First thing to check is that docker socket is correctly mounted and permissions are in place
We are using Rancher to orchestrate, but basically it comes down to :
docker run -v /var/run/docker.sock:/var/run/docker.sock:rw -u root -e OUTPUT_URL=http://xxxxxx:9200 <custom-image>
Custom Dockerfile :
COPY metricbeat.yml /usr/share/metricbeat/metricbeat.yml
# Customized template to index rancher container labels
COPY template-5.6.2.json /usr/share/metricbeat/template-5.6.2.json
RUN chown -R root.root /usr/share/metricbeat
- module: docker
metricsets: ["container", "cpu", "diskio", "info", "memory", "network"]
When attaching to the container in a /bin/bash , I can use netcat to talk in the socket.
I can also see some HTTP requests/answers in the socket by strace'ing the metricbeat process
It's weird that you see requests/responses while stracing, but the client fails with i/o, I'm wondering if this may be a version incompatibility issue due to the client being too old?
Could you paste the responses you see?
I just checked and we are at
v17.05.0-ce version of the client so I'm not sure that's the issue, could you please try to run metricbeat from the host machine? instead of running it from a container. Just want to discard any issue with the socket
I installed metricbeat on the host with the same template/configuration to test it out and I do get the same error :
17/10/11 14:05:50.891976 docker.go:132: ERR An error occurred while getting docker stats: read unix @->/var/run/docker.sock: i/o timeout
We have scheduled to upgrade docker to 17.06-ce in the weeks to come, so we'll be able to test that.
docker info :
Server Version: 1.12.6
Storage Driver: btrfs
Build Version: Btrfs v4.4.1
Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Volume: local rancher-secrets
Network: host null bridge overlay
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.57-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
Total Memory: 7.793 GiB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
WARNING: bridge-nf-call-ip6tables is disabled
It seems this error is gone since we updated docker to
So that may indeed be an incompatibility issue between metricbeat
5.6.2 and docker
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.