# Docker Autodiscover of ElasticSearch container

**URL:** <https://discuss.elastic.co/t/docker-autodiscover-of-elasticsearch-container/227628>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 11, 2020, 7:04pm UTC](https://discuss.elastic.co/t/docker-autodiscover-of-elasticsearch-container/227628 "2020-04-11T19:04:53Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![LenaSikirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lenasikirin/32/65074_2.png) [@LenaSikirin](https://discuss.elastic.co/u/LenaSikirin)\
**Post date:** [April 11, 2020, 7:04pm UTC](https://discuss.elastic.co/t/docker-autodiscover-of-elasticsearch-container/227628/1 "2020-04-11T19:04:53Z")

</div>

Hi,

I'm trying to ship elasticsearch logs using filebeat's autodiscover feature + container input, but don't know how to split different types of es logs properly.  
For nginx - I can split log easily just by using [stream](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-container.html#_stream) field, but for elasticsearch it is always equals `stdout` for all 4 types of logs/assets and `actual` split field is under path `log.type` .

One example of docker log:

> ```
> {
> "log": { 
> "type": "server",
> "timestamp": "20xx-04-11T18:20:54,430Z",
> "level": "DEBUG",
> "component": "o.e.a.a.c.n.t.c.TransportCancelTasksAction",
> "cluster.name": "some-cluster",
> "node.name": "some-xxxxx",
> "message": "Removing ban for the parent [xxxxxx:xxxx] on the node [xxxxx-xxxx]",
> "cluster.uuid": "xxx-xxxx",
> "node.id": "xxxx-xxxx"  
> },
> "stream":"stdout",
> "time":"20xx-04-11T18:11:54.43434Z"
> }
> 
> ```

Here is my `bad` config file:

> ```
> filebeat.autodiscover:
> providers:
> - type: docker
> templates:
> # ElasticSearch services
> - condition.contains:
> docker.container.image: elasticsearch
> config:
> - module: elasticsearch
> server:
> input:
> type: container
> paths:
> - /var/lib/docker/containers/${data.docker.container.id}/*.log
> gc:
> input:
> type: container
> paths:
> - /var/lib/docker/containers/${data.docker.container.id}/*.log
> audit:
> input:
> type: container
> paths:
> - /var/lib/docker/containers/${data.docker.container.id}/*.log
> slowlog:
> input:
> type: container
> paths:
> - /var/lib/docker/containers/${data.docker.container.id}/*.log
> deprecation:
> input:
> type: container
> paths:
> - /var/lib/docker/containers/${data.docker.container.id}/*.log
> 
> ```

By using this config - filebeat can ship all of log types with proper pipeline, but it will lead to lots of weird duplicates (because gc pipeline know nothing about slowlog and vice versa).

How can I split different types of logs like [stream](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-container.html#_stream) field? (I can add logstash as middleware and depends on `log.type` specify different pipelines manually, but it looks so weird ☹ )

Best regards.

---

<div class="post-metadata">

**Author:** ![LenaSikirin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lenasikirin/32/65074_2.png) [@LenaSikirin](https://discuss.elastic.co/u/LenaSikirin)\
**Post date:** [April 13, 2020, 12:18am UTC](https://discuss.elastic.co/t/docker-autodiscover-of-elasticsearch-container/227628/2 "2020-04-13T00:18:06Z")

</div>

It looks I don't suppose I have a choice in all this. The only solution - to set proper pipeline manually by using logstash. :cry

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 11, 2020, 12:18am UTC](https://discuss.elastic.co/t/docker-autodiscover-of-elasticsearch-container/227628/3 "2020-05-11T00:18:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.