Docker autodiscover with default template harvests logs multiple times

[FSeidinger (Frank Seidinger)]

I tried the autodiscover with docker containers only, like described in https://github.com/elastic/beats/issues/6084 and it fails with parsing docker log files multiple times.

Consider the following config:

filebeat.autodiscover:
  providers:
    # Provider for our docker containers
    - type: docker
      templates:
        # Template for the spring boot json logging containers
        - condition:
            contains:
              docker.container.image: myuser/myimage
          config:
            - type: docker
              containers:
                ids:
                  - ${data.docker.container.id}
              encoding: utf-8
              json:
                keys_under_root: true
                add_error_key: true
                message_key: "message"
                overwrite_keys: true
                match: after
              fields:
                log.format.content: "json"
                log.format.layout: "spring-boot"
        # Template for all other containers
        - condition:
          config:
            - type: docker
              containers:
                ids:
                  - ${data.docker.container.id}
              encoding: utf-8
              fields:
                log.format.content: "plain"
                log.format.layout: "spring"

When I check this configuration with filebeat 6.6.2, it tells me that the config is ok. When I start this configuration, my expected behavior is:

1. The log from the container myuser/myimage is using the template for json logging
2. All other containers are using the default template

What happens is:

1. The log from the container myuser/myimage is harvested using the given template
2. The log of all containers including myuser/myimage is harvested using the default template.

Therefore the log for the container using image myuser/myimage is harvested twice.

Is this the expected behavior? And if so, can the second log stream be suppressed in any way?

[kvch (Noémi Ványi)]

Your first configuration block for myuser/myimage seems incorrect. Our config testing tools is basic and is not able to detect all errors right now.

The configuration group json does not have an option named match. See: https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-docker.html#filebeat-input-docker-config-json
Also, I think you don't need json.* options at all, as Filebeat parses JSON Docker logs for you. Is there any reason you need it?

Furthermore, you need to make sure that the second condition is properly set up. Otherwise, Filebeat tries to open the same file from different inputs which leads to problems. So you need to filter out the container logs from the previous input.
For example:

- condition:
    contains.not:
      docker.container.image: myuser/myimage