OS: CentOS 7, kernel 3.10
Docker version : 1.12.6
Elastic version : 5.6.2
Metricbeat version : 5.6.2
Hi,
I'm trying to run metricbeat as a docker container to monitor other docker containers.
As I don't have the same docker group id across my environments i can't usermod -g metricbeat so that metricbeat can communicate through the docker socket.
I chose to run the process as root inside the container.
The container is privileged, and has rw access to the /var/run/docker.sock mounted as a volume (tested with netcat)
The error I get is : ... docker.go:132: ERR An error occurred while getting docker stats: read unix @->/var/run/docker.sock: i/o timeout
It seems that all data is pushed to ES correctly despite this error.
Could you share your metricbeat.yml and the docker command you are using? First thing to check is that docker socket is correctly mounted and permissions are in place
We are using Rancher to orchestrate, but basically it comes down to : docker run -v /var/run/docker.sock:/var/run/docker.sock:rw -u root -e OUTPUT_URL=http://xxxxxx:9200 <custom-image>
Custom Dockerfile :
COPY metricbeat.yml /usr/share/metricbeat/metricbeat.yml
# Customized template to index rancher container labels
COPY template-5.6.2.json /usr/share/metricbeat/template-5.6.2.json
USER root
RUN chown -R root.root /usr/share/metricbeat
When attaching to the container in a /bin/bash , I can use netcat to talk in the socket.
I can also see some HTTP requests/answers in the socket by strace'ing the metricbeat process
It's weird that you see requests/responses while stracing, but the client fails with i/o, I'm wondering if this may be a version incompatibility issue due to the client being too old?
I just checked and we are at v17.05.0-ce version of the client so I'm not sure that's the issue, could you please try to run metricbeat from the host machine? instead of running it from a container. Just want to discard any issue with the socket
It seems this error is gone since we updated docker to 17.06-ce.
So that may indeed be an incompatibility issue between metricbeat 5.6.2 and docker 1.12.6
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.