Filebeat 7.3.0 Autodiscover configuration introduction yields "Cannot connect to the Docker daemon" error

[nmccoy (nathan)]

I would like to roll out an ELK stack using filebeat to transport logs from custom docker containers. The OS is Ubuntu 18.04.

I am trying to do this by using the filebeat autodiscover feature by following the documentation:

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html

In the documentation is states that you should just add a filebeat.autodiscover field to the yaml file.

Here is what I have done:

1. Run the docker container and copy the default config
2. Add in the filebeat.autodiscover in the documentation

If I run with the filebeat.yaml file containing the filebeat.autodiscover field, it does not run and gives the following error:

2019-08-06T09:46:40.913Z	ERROR	instance/beat.go:877	Exiting: error in autodiscover provider settings: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Exiting: error in autodiscover provider settings: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

Here is my filebeat.yaml config file:

filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      labels.dedot: true
      templates:
        - condition:
            contains:
              docker.container.image: redis
          config:
            - type: docker
              containers.ids:
                - "${data.docker.container.id}"
              exclude_lines: ["^\\s+[\\-`('.|_]"]  # drop asciiart lines

processors:
- add_cloud_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:elasticsearch:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

And i am running the command:

$ docker run -v ~/elastic/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:7.3.0

where ~/elastic/filebeat/filebeat.yml contains the config

When I remove the filebeat.autodiscover section, the config runs without the error.

Thank you for any suggestions

[Michal_Pristas]

is user used to run filebeat part of your docker group?

[nmccoy (nathan)]

I dont specify a --user argument to docker, just the command posted above.
Is this necessary, and if so why?

[Michal_Pristas]

@nmccoy

Cannot connect to the Docker daemon

my thinking was that filebeat is not able to connect to docker daemon to subscribe for events.
if this is the case, please check if

• docker is running
• user which is used for running filebeat has access rights for /var/run/docker.sock

usually /var/run/docker.sock is accessible for owner and a group where owner is a root and group set to docker. check whether user used to run filebeat is member of a docker group