Docker unix socket i/o timeout error despite correctly working

OS: CentOS 7, kernel 3.10
Docker version : 1.12.6
Elastic version : 5.6.2
Metricbeat version : 5.6.2

Hi,
I'm trying to run metricbeat as a docker container to monitor other docker containers.
As I don't have the same docker group id across my environments i can't usermod -g metricbeat so that metricbeat can communicate through the docker socket.
I chose to run the process as root inside the container.
The container is privileged, and has rw access to the /var/run/docker.sock mounted as a volume (tested with netcat)

The error I get is :
... docker.go:132: ERR An error occurred while getting docker stats: read unix @->/var/run/docker.sock: i/o timeout

It seems that all data is pushed to ES correctly despite this error.

After testing, it seems that these errors occur despite sending complete data to elasticsearch, and using debug doesn't change anything.

updating first post.

Hi @kwaio,

Could you share your metricbeat.yml and the docker command you are using? First thing to check is that docker socket is correctly mounted and permissions are in place

Hi,

We are using Rancher to orchestrate, but basically it comes down to :
docker run -v /var/run/docker.sock:/var/run/docker.sock:rw -u root -e OUTPUT_URL=http://xxxxxx:9200 <custom-image>

Custom Dockerfile :

COPY metricbeat.yml /usr/share/metricbeat/metricbeat.yml
# Customized template to index rancher container labels
COPY template-5.6.2.json /usr/share/metricbeat/template-5.6.2.json
USER root
RUN chown -R root.root /usr/share/metricbeat

Metricbeat.yml :

  hosts: ["${OUTPUT_URL}"]
  index: "metricbeat-%{+yyyy.MM.dd}"
  template.enabled: true
  template.path: "template-5.6.2.json"
  template.overwrite: true
metricbeat.modules:
- module: docker
  metricsets: ["container", "cpu", "diskio", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  enabled: true
  period: 10s

When attaching to the container in a /bin/bash , I can use netcat to talk in the socket.
I can also see some HTTP requests/answers in the socket by strace'ing the metricbeat process

It's weird that you see requests/responses while stracing, but the client fails with i/o, I'm wondering if this may be a version incompatibility issue due to the client being too old?

Could you paste the responses you see?

I just checked and we are at v17.05.0-ce version of the client so I'm not sure that's the issue, could you please try to run metricbeat from the host machine? instead of running it from a container. Just want to discard any issue with the socket

I installed metricbeat on the host with the same template/configuration to test it out and I do get the same error :

17/10/11 14:05:50.891976 docker.go:132: ERR An error occurred while getting docker stats: read unix @->/var/run/docker.sock: i/o timeout

We have scheduled to upgrade docker to 17.06-ce in the weeks to come, so we'll be able to test that.

docker info :

Containers: 20
 Running: 19
 Paused: 0
 Stopped: 1
Images: 20
Server Version: 1.12.6
Storage Driver: btrfs
 Build Version: Btrfs v4.4.1
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local rancher-secrets
 Network: host null bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.57-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.793 GiB
Name: evx4405463
ID: 3H5E:22JL:3564:BYRM:CNNW:3GYH:Z3HL:5NOL:RHYD:D25H:4U43:BWXV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Hi,

It seems this error is gone since we updated docker to 17.06-ce.
So that may indeed be an incompatibility issue between metricbeat 5.6.2 and docker 1.12.6

thanks

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.