# Docker autodiscover with default template harvests logs multiple times

**URL:** <https://discuss.elastic.co/t/docker-autodiscover-with-default-template-harvests-logs-multiple-times/173490>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [March 22, 2019, 12:19pm UTC](https://discuss.elastic.co/t/docker-autodiscover-with-default-template-harvests-logs-multiple-times/173490 "2019-03-22T12:19:45Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![FSeidinger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fseidinger/32/42591_2.png) [@FSeidinger](https://discuss.elastic.co/u/FSeidinger)\
**Post date:** [March 22, 2019, 12:19pm UTC](https://discuss.elastic.co/t/docker-autodiscover-with-default-template-harvests-logs-multiple-times/173490/1 "2019-03-22T12:19:45Z")

</div>

I tried the autodiscover with docker containers only, like described in [https://github.com/elastic/beats/issues/6084](https://github.com/elastic/beats/issues/6084) and it fails with parsing docker log files multiple times.

Consider the following config:

```auto
filebeat.autodiscover:
  providers:
    # Provider for our docker containers
    - type: docker
      templates:
        # Template for the spring boot json logging containers
        - condition:
            contains:
              docker.container.image: myuser/myimage
          config:
            - type: docker
              containers:
                ids:
                  - ${data.docker.container.id}
              encoding: utf-8
              json:
                keys_under_root: true
                add_error_key: true
                message_key: "message"
                overwrite_keys: true
                match: after
              fields:
                log.format.content: "json"
                log.format.layout: "spring-boot"
        # Template for all other containers
        - condition:
          config:
            - type: docker
              containers:
                ids:
                  - ${data.docker.container.id}
              encoding: utf-8
              fields:
                log.format.content: "plain"
                log.format.layout: "spring"

```

When I check this configuration with filebeat 6.6.2, it tells me that the config is ok. When I start this configuration, my expected behavior is:

1. The log from the container `myuser/myimage` is using the template for json logging
2. All other containers are using the default template

What happens is:

1. The log from the container `myuser/myimage` is harvested using the given template
2. The log of all containers including `myuser/myimage` is harvested using the default template.

Therefore the log for the container using image `myuser/myimage` is harvested twice.

Is this the expected behavior? And if so, can the second log stream be suppressed in any way?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [March 25, 2019, 10:25am UTC](https://discuss.elastic.co/t/docker-autodiscover-with-default-template-harvests-logs-multiple-times/173490/2 "2019-03-25T10:25:09Z")

</div>

Your first configuration block for `myuser/myimage` seems incorrect. Our config testing tools is basic and is not able to detect all errors right now.

The configuration group `json` does not have an option named `match`. See: [https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-docker.html#filebeat-input-docker-config-json](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-docker.html#filebeat-input-docker-config-json)  
Also, I think you don't need `json.*` options at all, as Filebeat parses JSON Docker logs for you. Is there any reason you need it?

Furthermore, you need to make sure that the second condition is properly set up. Otherwise, Filebeat tries to open the same file from different inputs which leads to problems. So you need to filter out the container logs from the previous input.  
For example:

```auto
- condition:
    contains.not:
      docker.container.image: myuser/myimage

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 22, 2019, 10:25am UTC](https://discuss.elastic.co/t/docker-autodiscover-with-default-template-harvests-logs-multiple-times/173490/3 "2019-04-22T10:25:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.