# Docker unix socket i/o timeout error despite correctly working

**URL:** <https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [October 10, 2017, 9:37am UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333 "2017-10-10T09:37:09Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![kwaio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwaio/32/22922_2.png) [@kwaio](https://discuss.elastic.co/u/kwaio)\
**Post date:** [October 10, 2017, 9:37am UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333/1 "2017-10-10T09:37:09Z")

</div>

OS: CentOS 7, kernel 3.10  
Docker version : 1.12.6  
Elastic version : 5.6.2  
Metricbeat version : 5.6.2

Hi,  
I'm trying to run metricbeat as a docker container to monitor other docker containers.  
As I don't have the same docker group id across my environments i can't `usermod -g metricbeat` so that metricbeat can communicate through the docker socket.  
I chose to run the process as root inside the container.  
The container is privileged, and has rw access to the /var/run/docker.sock mounted as a volume (tested with netcat)

The error I get is :  
`... docker.go:132: ERR An error occurred while getting docker stats: read unix @->/var/run/docker.sock: i/o timeout`

It seems that all data is pushed to ES correctly despite this error.

---

<div class="post-metadata">

**Author:** ![kwaio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwaio/32/22922_2.png) [@kwaio](https://discuss.elastic.co/u/kwaio)\
**Post date:** [October 10, 2017, 3:11pm UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333/2 "2017-10-10T15:11:21Z")

</div>

After testing, it seems that these errors occur despite sending complete data to elasticsearch, and using debug doesn't change anything.

updating first post.

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [October 10, 2017, 3:34pm UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333/3 "2017-10-10T15:34:04Z")

</div>

Hi @kwaio,

Could you share your metricbeat.yml and the docker command you are using? First thing to check is that docker socket is correctly mounted and permissions are in place

---

<div class="post-metadata">

**Author:** ![kwaio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwaio/32/22922_2.png) [@kwaio](https://discuss.elastic.co/u/kwaio)\
**Post date:** [October 10, 2017, 3:55pm UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333/4 "2017-10-10T15:55:21Z")

</div>

Hi,

We are using Rancher to orchestrate, but basically it comes down to :  
`docker run -v /var/run/docker.sock:/var/run/docker.sock:rw -u root -e OUTPUT_URL=http://xxxxxx:9200 <custom-image>`

Custom Dockerfile :

```auto
COPY metricbeat.yml /usr/share/metricbeat/metricbeat.yml
# Customized template to index rancher container labels
COPY template-5.6.2.json /usr/share/metricbeat/template-5.6.2.json
USER root
RUN chown -R root.root /usr/share/metricbeat

```

Metricbeat.yml :

```auto
  hosts: ["${OUTPUT_URL}"]
  index: "metricbeat-%{+yyyy.MM.dd}"
  template.enabled: true
  template.path: "template-5.6.2.json"
  template.overwrite: true
metricbeat.modules:
- module: docker
  metricsets: ["container", "cpu", "diskio", "info", "memory", "network"]
  hosts: ["unix:///var/run/docker.sock"]
  enabled: true
  period: 10s

```

When attaching to the container in a /bin/bash , I can use netcat to talk in the socket.  
I can also see some HTTP requests/answers in the socket by strace'ing the metricbeat process

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [October 11, 2017, 1:45pm UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333/5 "2017-10-11T13:45:33Z")

</div>

It's weird that you see requests/responses while stracing, but the client fails with i/o, I'm wondering if this may be a version incompatibility issue due to the client being too old?

Could you paste the responses you see?

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [October 11, 2017, 2:01pm UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333/6 "2017-10-11T14:01:23Z")

</div>

I just checked and we are at `v17.05.0-ce` version of the client so I'm not sure that's the issue, could you please try to run metricbeat from the host machine? instead of running it from a container. Just want to discard any issue with the socket

---

<div class="post-metadata">

**Author:** ![kwaio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwaio/32/22922_2.png) [@kwaio](https://discuss.elastic.co/u/kwaio)\
**Post date:** [October 11, 2017, 2:07pm UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333/7 "2017-10-11T14:07:41Z")

</div>

I installed metricbeat on the host with the same template/configuration to test it out and I do get the same error :

```auto
17/10/11 14:05:50.891976 docker.go:132: ERR An error occurred while getting docker stats: read unix @->/var/run/docker.sock: i/o timeout

```

We have scheduled to upgrade docker to 17.06-ce in the weeks to come, so we'll be able to test that.

`docker info` :

```auto
Containers: 20
 Running: 19
 Paused: 0
 Stopped: 1
Images: 20
Server Version: 1.12.6
Storage Driver: btrfs
 Build Version: Btrfs v4.4.1
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local rancher-secrets
 Network: host null bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 4.4.57-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.793 GiB
Name: evx4405463
ID: 3H5E:22JL:3564:BYRM:CNNW:3GYH:Z3HL:5NOL:RHYD:D25H:4U43:BWXV
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

```

---

<div class="post-metadata">

**Author:** ![kwaio](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kwaio/32/22922_2.png) [@kwaio](https://discuss.elastic.co/u/kwaio)\
**Post date:** [November 2, 2017, 8:43am UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333/8 "2017-11-02T08:43:31Z")

</div>

Hi,

It seems this error is gone since we updated docker to `17.06-ce`.  
So that may indeed be an incompatibility issue between metricbeat `5.6.2` and docker `1.12.6`

thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 30, 2017, 8:43am UTC](https://discuss.elastic.co/t/docker-unix-socket-i-o-timeout-error-despite-correctly-working/103333/9 "2017-11-30T08:43:41Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.