# Autodiscovery with modules in Docker
**URL:** https://discuss.elastic.co/t/autodiscovery-with-modules-in-docker/176390
**Category:** Beats
**Tags:** filebeat
**Created:** [April 11, 2019, 11:21am UTC](https://discuss.elastic.co/t/autodiscovery-with-modules-in-docker/176390 "2019-04-11T11:21:53Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@ftec](https://discuss.elastic.co/u/ftec)
#### Post date: [April 11, 2019, 11:21am UTC](https://discuss.elastic.co/t/autodiscovery-with-modules-in-docker/176390/1 "2019-04-11T11:21:53Z")
Hi,
I have the deployed ES stack (ES + Kibana + Metricbeat + Filebeat) to a Docker Swarm stack.
That stack contains also some services like MySQL (mariadb), Kafka+Zookeeper, Traefik gateways, etc.
I want to read Docker logs via Filebeat and ingest them in ES.
Since Filebeat has the very nice "modules" feature for the above mentioned tools I wanted to put them on use.
But I didn't managed to do it on Docker.
I map these volumes to Filebeat container:
- '/var/run/docker.sock:/var/run/docker.sock:rw'
- '/var/lib/docker/containers:/usr/share/filebeat/source-logs:rw'
This is my filebeat.yml - the problem is that autodiscovery doesn't enable the modules:
```auto
name: ${SHIPPER_NAME:"filebeat"}
logging.level: info
logging.metrics.enabled: true
logging.metrics.period: 60s
logging.to_syslog: false
logging.json: false
filebeat.shutdown_timeout: 5
queue.mem:
events: 8192
flush.min_events: 0
flush.timeout: 0s
filebeat.config:
prospectors:
enabled: true
path: prospectors.d/*.yml
reload.enabled: true
reload.period: 10s
modules:
enabled: true
path: modules.d/*.yml
reload.enabled: true
reload.period: 10s
setup.template.enabled: true
setup.template.overwrite: false
setup.template.fields: "fields.yml"
setup.template.name: ${ES_TEMPLATE}
setup.template.pattern: ${ES_TEMPLATE}-*
setup.template.settings:
index.number_of_shards: 1
index.number_of_replicas: 1
setup.dashboards.enabled: true
setup.dashboards.directory: ${FILEBEAT_HOME}/logs_dashboards
setup.dashboards.beat: filebeat
setup.dashboards.kibana_index: .kibana
setup.dashboards.index: ${ES_TEMPLATE}-*
setup.kibana:
host: '${KIBANA_URL}'
output.elasticsearch:
enabled: true
max_retries: 10
timeout: 90
loadbalance: true
worker: 2
hosts: '${ELASTICSEARCH_URL}'
index: '${ES_INDEX}'
pipeline: '${ES_PIPELINE:""}'
# filebeat.modules:
# - module: mysql
# error:
# enabled: true
# slowlog:
# enabled: true
# - module: traefik
# access:
# enabled: true
# - module: kafka
# log:
# enabled: true
filebeat.autodiscover:
providers:
- type: docker
templates:
- condition:
contains:
docker.container.image: "mariadb"
config:
- module: mysql
slowlog:
enabled: true
var.paths:
- /usr/share/filebeat/source-logs/${data.docker.container.id}/*.log
input:
type: docker
containers.ids:
- "${data.docker.container.id}"
error:
enabled: true
var.paths:
- /usr/share/filebeat/source-logs/${data.docker.container.id}/*.log
input:
type: docker
containers.ids:
- "${data.docker.container.id}"
- condition:
contains:
docker.container.image: "traefik"
config:
- module: traefik
access:
enabled: true
var.paths:
- /usr/share/filebeat/source-logs/${data.docker.container.id}/*.log
input:
type: docker
containers.ids:
- "${data.docker.container.id}"
- condition:
contains:
docker.container.image: "johnnypark/kafka-zookeeper"
config:
- module: kafka
log:
enabled: true
var.paths:
- /usr/share/filebeat/source-logs/${data.docker.container.id}/*.log
input:
type: docker
containers.ids:
- "${data.docker.container.id}"
- condition:
contains:
docker.container.image: my_custom_services_image
config:
- type: docker
containers.path: "/usr/share/filebeat/source-logs"
containers.ids: ["${data.docker.container.id}"]
multiline.pattern: '${MULTILINE_PATTERN:""}'
multiline.negate: '${MULTILINE_NEGATE_PATTERN:false}'
multiline.match: '${MULTILINE_MATCH_POSITION:"after"}'
filebeat.prospectors:
- type: docker
enabled: true
containers.path: "/usr/share/filebeat/source-logs"
containers.ids: ["*"]
multiline.pattern: '${MULTILINE_PATTERN:""}'
multiline.negate: '${MULTILINE_NEGATE_PATTERN:false}'
multiline.match: '${MULTILINE_MATCH_POSITION:"after"}'
processors:
- add_locale:
format: offset
- add_cloud_metadata: ~
- add_docker_metadata:
host: '${DOCKER_ENDPOINT:unix:///var/run/docker.sock}'
```
---
### Author:  [@pierhugues](https://discuss.elastic.co/u/pierhugues)
#### Post date: [April 11, 2019, 12:54pm UTC](https://discuss.elastic.co/t/autodiscovery-with-modules-in-docker/176390/2 "2019-04-11T12:54:50Z")
Looking at the config that should work, do you have any related message in the logs?
---
### Author:  [@ftec](https://discuss.elastic.co/u/ftec)
#### Post date: [April 12, 2019, 8:19am UTC](https://discuss.elastic.co/t/autodiscovery-with-modules-in-docker/176390/3 "2019-04-12T08:19:57Z")
The logs messages are put in filebeat\* index (with metadata fields from docker processor) but without any specific modules processing.
It looks to me that those autodiscover events are not triggered.. the "contains: docker.container.image" maybe is not triggered?
Since you say that the config looks ok, how can I check if those conditions/events works - there are some specific messages in filebeat logs or so?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [May 10, 2019, 8:20am UTC](https://discuss.elastic.co/t/autodiscovery-with-modules-in-docker/176390/4 "2019-05-10T08:20:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.