Filebeat Autodiscovery: Different configurations for different Docker containers

Elastic Stack Beats
2

I think the problem there is that there are multiple inputs monitoring the same file. In your first template you are configuring a path for a single container:

              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log

And in the second template you are configuring a path for all the containers:

              paths:
                - /var/lib/docker/containers/*/*.log

This configuration will be created for each docker that doesn't contains the alpha container name, instantiating many inputs with the same configuration.

I think that you should use the same path in both templates, take into account that it will use a different container id each time it is instantiated:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.name: alpha
          config:
            - type: log
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log

              multiline.pattern: '^({"log":")[0-9]{4}-[0-9]{2}-[0-9]{2}.{85}(: Mosaic response Xml)'
              multiline.negate: true
              multiline.match: after
              multiline.flush_pattern: '^({"log":",{Date=\[)[A-Za-z]{3,5},.{4}[A-Za-z]{3,5}[[:blank:]].{20}(Server)'

              fields_under_root: true
              fields:
                appid: id

        - condition:
            not.contains:
              docker.container.name: alpha
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log

              fields_under_root: true
              fields:
                appid: id

output.file:
  path: "/var/lib/docker/filebeat"

processors:
  - add_docker_metadata: ~

Let's see if this works.

If you are going to have more different configurations, you may consider using hints-based autodiscover, that allows you to define a default configuration that can be overriden per pod or per container using annotations.

1 Like