# Filebeat: docker autodiscover - do I have things right?
**URL:** https://discuss.elastic.co/t/filebeat-docker-autodiscover-do-i-have-things-right/138404
**Category:** Beats
**Tags:** filebeat
**Created:** [July 3, 2018, 1:47pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscover-do-i-have-things-right/138404 "2018-07-03T13:47:55Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)
#### Post date: [July 3, 2018, 1:47pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscover-do-i-have-things-right/138404/1 "2018-07-03T13:47:55Z")
Hi there,
After reading and watching this information:
> **[Docker and Kubernetes Hints-Based Autodiscover with Beats](https://www.elastic.co/blog/docker-and-kubernetes-hints-based-autodiscover-with-beats)**
>
> Beats 6.3 brings new features to do Kubernetes and Docker autodiscovery. You can now define logging & monitoring settings using labels or annotations.
[https://www.elastic.co/guide/en/beats/filebeat/6.3/configuration-autodiscover.html](https://www.elastic.co/guide/en/beats/filebeat/6.3/configuration-autodiscover.html)
[https://www.elastic.co/webinars/elasticsearch-log-collection-with-kubernetes-docker-and-containers](https://www.elastic.co/webinars/elasticsearch-log-collection-with-kubernetes-docker-and-containers)
> **[Monitoring an application running in Docker containers and Kubernetes with the...](https://www.elastic.co/blog/monitoring-an-application-running-in-docker-containers-and-kubernetes-with-the-elastic-stack)**
>
> An introduction to the benefits of Elastic modules and autodiscover feature when monitoring the logs and metrics from a Docker container environment. Collect, search, and visualize the logs and metrics from your application, Docker, and Kubernetes...
I'm still not 100% certain that I have the right filebeat.yml for my use case. I have 7 "static" docker containers and a few that start and stop. I'm trying to monitor all the logs (and performance metrics) from these containers automatically using filebeat "autodiscover". One of my containers is the official Elasticsearch docker container, so if I can enhance anything by using the "elasticsearch" module, that would be cool.
```auto
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
223b4e9e9203 docker.elastic.co/beats/filebeat:6.3.0 "/usr/local/bin/dock…" About an hour ago Up 6 minutes filebeat
ac124ea897ec traceloggercisco/tacassist:latest "/bin/sh -c 'cron &&…" 12 days ago Up 25 hours 80/tcp, 85/tcp, 0.0.0.0:85->443/tcp tacassist
02f08fe30ed9 traceloggercisco/maintenance:latest "crond -f" 12 days ago Up 25 hours maintenance
3ec9b5d75b56 traceloggercisco/idm:latest "python3 ./main.py" 12 days ago Up 25 hours 0.0.0.0:8506->80/tcp idm
2f9c2b74197c traceloggercisco/logstorage:latest "python3 ./LogStorag…" 12 days ago Up 25 hours logstorage
bd4b65647675 traceloggercisco/webapp:latest "/entrypoint.sh /usr…" 12 days ago Up 25 hours 80/tcp, 0.0.0.0:8443->443/tcp webapp
1db10faa880b elasticsearch:tracelogger "/bin/bash bin/es-do…" 3 weeks ago Up 25 hours 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp elasticsearch
```
my filebeat.yml is:
```auto
filebeat.prospectors:
- type: docker
containers.ids:
- '*'
#=========================== Filebeat autodiscover ==============================
filebeat.autodiscover:
# Autodiscover docker containers and parse logs
providers:
- type: docker
templates:
- condition.contains:
docker.container.image: elasticsearch
config:
- module: elasticsearch
log:
input:
type: docker
containers.ids:
- "${data.docker.container.id}"
processors:
- add_docker_metadata:
```
Does this look alright? Do I have everything I need? I am seeing logs coming in, but not from all the containers. I'm assuming that filebeat should find all the containers and retrieve their logs even if filebeat has started after the other containers?
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [July 3, 2018, 2:19pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscover-do-i-have-things-right/138404/2 "2018-07-03T14:19:05Z")
There is no Elasticsearch filebeat module in 6.3, it will only be available in 6.4. Might this be the issue?
Where did you stumble over the Elasticsearch Filebeat module? Mainly curious to figure out if we have it in our communication somewhere where it shouldn't be for 6.3.
---
### Author:  [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)
#### Post date: [July 4, 2018, 12:33pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscover-do-i-have-things-right/138404/3 "2018-07-04T12:33:16Z")
filebeat 6.3 documentation ([https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html)) says it supports modules (gives redis as an example though). I assumed that autodiscover supported all existing modules. Might be good to clarify that in the docs.
So what you are saying is that filebeat6.3 won't do any special parsing of the elasticsearch logs that are in my docker container. However will it still extract those container logs like any generic container using the "docker logs " API?
What would the current config be then to get all the logs from existing and new docker containers?
```auto
filebeat.prospectors:
- type: docker
containers.ids:
- '*'
#=========================== Filebeat autodiscover ==============================
filebeat.autodiscover:
# Autodiscover docker containers and parse logs
providers:
- type: docker
processors:
- add_docker_metadata:
```
Also, do I need both the filebeat.prospectors and filebeat.autodiscover config lines? I didn't see this clearly in the documentation and examples, but saw it in the webinar (which is excellent by the way!).
This webinar used the nginx module in the autodiscover example: [https://www.elastic.co/webinars/elasticsearch-log-collection-with-kubernetes-docker-and-containers](https://www.elastic.co/webinars/elasticsearch-log-collection-with-kubernetes-docker-and-containers)
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [July 4, 2018, 1:56pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscover-do-i-have-things-right/138404/4 "2018-07-04T13:56:23Z")
Autodiscovery supports all modules which were released with this version: [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules.html) Elasticsearch module was not released yet.
It is correct that you can still get all the Elasticsearch logs but you don't get the parsing like you stated above.
You should only need the autodiscovery part. It will pick up the logs from all running containers and new ones. Also the processor should not be needed for the docker metadata as autodiscovery will add the data automatically.
---
### Author:  [@sgreszcz](https://discuss.elastic.co/u/sgreszcz)
#### Post date: [July 4, 2018, 2:28pm UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscover-do-i-have-things-right/138404/5 "2018-07-04T14:28:17Z")
So really all I need (for now as of filebeat6.3) for docker logs is:
```auto
#=========================== Filebeat autodiscover ==============================
filebeat.autodiscover:
# Autodiscover docker containers and parse logs
providers:
- type: docker
```
I'll open another thread for metricbeats as I'm trying to do the same config there. Thank you so much for your help!
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [July 9, 2018, 8:44am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscover-do-i-have-things-right/138404/6 "2018-07-09T08:44:11Z")
Yes, that should do it. The part I'm not 100% sure about anymore is if you also need the
```auto
containers.ids:
- '*'
```
But I think it's the default if not configured so it should just work.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 6, 2018, 8:44am UTC](https://discuss.elastic.co/t/filebeat-docker-autodiscover-do-i-have-things-right/138404/7 "2018-08-06T08:44:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.