Docker unix socket i/o timeout error despite correctly working

Elastic Stack Beats
1

OS: CentOS 7, kernel 3.10
Docker version : 1.12.6
Elastic version : 5.6.2
Metricbeat version : 5.6.2

Hi,
I'm trying to run metricbeat as a docker container to monitor other docker containers.
As I don't have the same docker group id across my environments i can't usermod -g metricbeat so that metricbeat can communicate through the docker socket.
I chose to run the process as root inside the container.
The container is privileged, and has rw access to the /var/run/docker.sock mounted as a volume (tested with netcat)

The error I get is :
... docker.go:132: ERR An error occurred while getting docker stats: read unix @->/var/run/docker.sock: i/o timeout

It seems that all data is pushed to ES correctly despite this error.

2

After testing, it seems that these errors occur despite sending complete data to elasticsearch, and using debug doesn't change anything.

updating first post.

3

Hi @kwaio,

Could you share your metricbeat.yml and the docker command you are using? First thing to check is that docker socket is correctly mounted and permissions are in place

4

Hi,

We are using Rancher to orchestrate, but basically it comes down to :
docker run -v /var/run/docker.sock:/var/run/docker.sock:rw -u root -e OUTPUT_URL=http://xxxxxx:9200 <custom-image>

Custom Dockerfile :

COPY metricbeat.yml /usr/share/metricbeat/metricbeat.yml
# Customized template to index rancher container labels
COPY template-5.6.2.json /usr/share/metricbeat/template-5.6.2.json
USER root
RUN chown -R root.root /usr/share/metricbeat

Metricbeat.yml :

  hosts: ["${OUTPUT_URL}"]
  index: "metricbeat-%{+yyyy.MM.dd}"
  template.enabled: true
  template.path: "template-5.6.2.json"
  template.overwrite: true
metricbeat.modules:
- module: docker
  metricsets: ["container", "cpu", "diskio", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  enabled: true
  period: 10s

When attaching to the container in a /bin/bash , I can use netcat to talk in the socket.
I can also see some HTTP requests/answers in the socket by strace'ing the metricbeat process

5

It's weird that you see requests/responses while stracing, but the client fails with i/o, I'm wondering if this may be a version incompatibility issue due to the client being too old?

Could you paste the responses you see?

6

I just checked and we are at v17.05.0-ce version of the client so I'm not sure that's the issue, could you please try to run metricbeat from the host machine? instead of running it from a container. Just want to discard any issue with the socket

7

I installed metricbeat on the host with the same template/configuration to test it out and I do get the same error :

17/10/11 14:05:50.891976 docker.go:132: ERR An error occurred while getting docker stats: read unix @->/var/run/docker.sock: i/o timeout

We have scheduled to upgrade docker to 17.06-ce in the weeks to come, so we'll be able to test that.

docker info :

Containers: 20
 Running: 19
 Paused: 0
 Stopped: 1
Images: 20
Server Version: 1.12.6
Storage Driver: btrfs
 Build Version: Btrfs v4.4.1
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local rancher-secrets
 Network: host null bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.57-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.793 GiB
Name: evx4405463
ID: 3H5E:22JL:3564:BYRM:CNNW:3GYH:Z3HL:5NOL:RHYD:D25H:4U43:BWXV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8
8

Hi,

It seems this error is gone since we updated docker to 17.06-ce.
So that may indeed be an incompatibility issue between metricbeat 5.6.2 and docker 1.12.6

thanks

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.