# Filebeat autodiscover for docker does not work on /var/lib/docker/containers
**URL:** https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766
**Category:** Beats
**Tags:** filebeat
**Created:** [December 6, 2018, 4:41pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766 "2018-12-06T16:41:10Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@RogerLapin](https://discuss.elastic.co/u/RogerLapin)
#### Post date: [December 6, 2018, 4:41pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766/1 "2018-12-06T16:41:11Z")
Hi,
I am quite puzzled about the autodiscover feature for "tea"ing docker logs.
This look quite useful, but despite reading the documentation and the few posts about it, I could not manage to have it fully work.
To be clear : I managed to get a copy of the logs from within /var/lib/docker/volumes/whatever, but I did not manage to perform the same with /var/lib/docker/containers
Filebeat runs as long as the elk stack on a swarm environnement.
They are running under their own network, thus are identified by the service name (_e.g._ logstash:5044)
Here's my config ::
```
filebeat.autodiscover:
providers:
- type: docker
templates:
# conditions are not compulsory. I have tried to remove them, but the problem remains
- condition.or:
- contains.docker.container.name: "apache"
- contains.docker.container.name: "nginx"
config:
- type: docker
containers.ids:
- ${data.docker.container.id}
multiline:
pattern: '^\[#|\d{4}'
negate: true
match: after
- type: log
paths: # various attempts
- /var/lib/docker/containers/${data.docker.container.id}/*.log # KO
- /var/lib/docker/containers/631c570fcd974c248142726cf9c4a1aa21d45cc7b9a39a0eed8dc73eeb41d1df/*.log #KO
- /var/lib/docker/containers/*/*.log #KO
- /var/lib/docker/volumes/${data.docker.container.name}/_data/localhost_access_log*.txt #OK
- /var/lib/docker/volumes/whoAmTest_tmpvol*/_data/*.log #OK
# I tried to add this later, according to a post I read here : https://discuss.elastic.co/t/problem-getting-autodiscover-docker-to-work-with-filebeat/144349/10
filebeat.inputs:
- type: docker
containers.ids:
- "*"
processors:
- add_docker_metadata:
host: "unix:///var/run/docker.sock"
output.logstash:
hosts: ["logstash:5044"]
bulk_max_size: 4096
```
I know some paths are not following the documentation guidelines, but there are here for the sole purpose of giving a few clues about the problem (I hope!).
I see no error on filebeat logs, nor on ES or logstash.
It's just not working for the logs from /var/lib/docker/containers/_/_.log!
Any idea / hint?
---
### Author:  [@trajano](https://discuss.elastic.co/u/trajano)
#### Post date: [December 14, 2018, 11:38pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766/2 "2018-12-14T23:38:23Z")
I have a similar issue, but I am new to setting this all up. Here's my filebeats.yml which I bake into the image
```
filebeat.config:
modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
filebeat.autodiscover:
providers:
- type: docker
hints.enabled: true
processors:
- drop_fields:
fields:
- "docker.containers.labels"
- add_host_metadata: ~
- add_cloud_metadata: ~
output.logstash:
hosts: "${BEATS_HOST_PORT:log:5044}"
logging.level: warning
```
The `drop fields` was from [Using AutoDiscover feature for Docker does not work when running in Swarm mode](https://discuss.elastic.co/t/using-autodiscover-feature-for-docker-does-not-work-when-running-in-swarm-mode/120342/3) but that didn't work for me either.
My docker-compose file has the following service
```
docker-beats:
image: trajano.net/docker-beats
volumes:
- /var/lib/docker/containers:/var/lib/docker/containers:ro
- /var/run/docker.sock:/var/run/docker.sock:ro
deploy:
mode: global
```
Unfortunately I just get all the logs with no real attribution except for the file beats server
---
### Author:  [@trajano](https://discuss.elastic.co/u/trajano)
#### Post date: [December 17, 2018, 1:30pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766/3 "2018-12-17T13:30:00Z")
Seems that in Graylog it requires `fields` to contain the data so I altered the filebeat to contain and logs start to appear now
```
processors:
- drop_event:
when:
contains:
docker.container.image: "trajano.net/docker-beats"
- rename:
fields:
- from: "docker.container.image"
to: "fields.docker_container_image"
- from: "docker.container.name"
#to: "fields.docker_container_name"
to: "fields.source"
- from: "docker.container.id"
to: "fields.docker_container_id"
- rename:
fields:
- from: "docker.container.labels.com.docker.compose.service"
to: "fields.docker_compose_service"
- from: "docker.container.labels.com.docker.compose.project"
to: "fields.docker_compose_project"
ignore_missing: true
- rename:
fields:
- from: "docker.container.labels.com.docker.swarm.node.id"
to: "fields.docker_swarm_node"
- from: "docker.container.labels.com.docker.swarm.task.name"
to: "fields.docker_swarm_task"
- from: "docker.container.labels.com.docker.swarm.service.name"
to: "fields.docker_swarm_service"
- from: "docker.container.labels.com.docker.stack.namespace"
to: "fields.docker_stack_namespace"
ignore_missing: true
```
However, I am expecting the labels to still trigger some change or initial parsing, but I can't seem to get that part working the labels in question are written as
```
web:
image: trajano/nginx-letsencrypt
deploy:
labels:
- co.elastic.logs/module=nginx
- co.elastic.logs/fileset.stdout=access
- co.elastic.logs/fileset.stderr=error
```
But looking through the debug logs it does not appear to do anything with it.
---
### Author:  [@pierhugues](https://discuss.elastic.co/u/pierhugues)
#### Post date: [December 17, 2018, 1:56pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766/4 "2018-12-17T13:56:06Z")
@RogerLapin
Sorry for the delay did you get it working? By default the docker input should pick up logs from `/var/lib/docker/containers`.
@trajano This look like another issue I would create a separate thread for that. Also if you drop the `"docker.containers.labels` you are probably dropping the attribution.
---
### Author:  [@trajano](https://discuss.elastic.co/u/trajano)
#### Post date: [December 17, 2018, 2:10pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766/5 "2018-12-17T14:10:03Z")
I actually got rid of
```auto
- drop_fields:
fields:
- "docker.containers.labels"
```
I'll spawn off a separate thread.
in my last version but comparing mine and the OP, I wonder if the OP added the `/var/lib/docker/containers/` volume
---
### Author:  [@RogerLapin](https://discuss.elastic.co/u/RogerLapin)
#### Post date: [December 17, 2018, 2:29pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766/6 "2018-12-17T14:29:27Z")
@pierhugues
No, I am still blind here : don't know why it's not working.
I forgot to mention elk version ; 6.4.2.
May it be related to the swarm mode?
---
### Author:  [@pierhugues](https://discuss.elastic.co/u/pierhugues)
#### Post date: [December 17, 2018, 4:37pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766/7 "2018-12-17T16:37:14Z")
@RogerLapin If you set debug log level do you get any errors in the Filebeat's logs?
---
### Author:  [@RogerLapin](https://discuss.elastic.co/u/RogerLapin)
#### Post date: [December 18, 2018, 3:04pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766/8 "2018-12-18T15:04:58Z")
@pierhugues I don't see any errors, or message that would indicate any problème with the autodiscover feature. It looks like it takes all the configured path into consideration as expected, and althouth it states that the feature is "beta", I see no related problem in the logs...
Though, the logs are especially verbose, and I am not sure what I should be looking for (but for obvious errors or warnings)
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [January 15, 2019, 3:04pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-for-docker-does-not-work-on-var-lib-docker-containers/159766/9 "2019-01-15T15:04:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.