Filebeat Autodiscovery: Different configurations for different Docker containers

Elastic Stack Beats
1

Hello ,

I'm currently drawing logs from around 30 Docker containers with a single input and it works perfectly. However, I have recently discovered that one of these containers exports XML which will need a separate configuration that utilises multi line message patterns (which I have already tested and got working). My issue now is getting two inputs to work alongside each other independently. I've recently found autodiscovery which lets me point a unique config at a single container which is great, but I ran into an error where Filebeat is trying to read the same containers more than once. So my main task is getting the rest of the config to work on the remaining 29 containers, but also getting it to ignore the XML container.

I also entertained the idea of possibly using autodiscover with a normal Filebeat input, but I again ran into the issue of getting the rest of the config to ignore the XML container as the container ID's are constantly changing so I cannot enter a specific path. Any help would be much appreciated.

Below is my current configuration.

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.name: alpha
          config:
            - type: log
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log

              multiline.pattern: '^({"log":")[0-9]{4}-[0-9]{2}-[0-9]{2}.{85}(: Mosaic response Xml)'
              multiline.negate: true
              multiline.match: after
              multiline.flush_pattern: '^({"log":",{Date=\[)[A-Za-z]{3,5},.{4}[A-Za-z]{3,5}[[:blank:]].{20}(Server)'

              fields_under_root: true
              fields:
                appid: id

        - condition:
            not.contains:
              docker.container.name: alpha
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/*/*.log

              exclude_files:
                - '^var/lib/docker/containers/${data.docker.container.id}/*.log'

              fields_under_root: true
              fields:
                appid: id

output.file:
  path: "/var/lib/docker/filebeat"

processors:
  - add_docker_metadata: ~

And here is the error message I am greeted with:

2020-05-05T15:40:11.482+0100    ERROR   [autodiscover]  cfgfile/list.go:96      
Error creating runner from config: Can only start an input when all related states 
are finished

Thanks,
Tom

2

I think the problem there is that there are multiple inputs monitoring the same file. In your first template you are configuring a path for a single container:

              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log

And in the second template you are configuring a path for all the containers:

              paths:
                - /var/lib/docker/containers/*/*.log

This configuration will be created for each docker that doesn't contains the alpha container name, instantiating many inputs with the same configuration.

I think that you should use the same path in both templates, take into account that it will use a different container id each time it is instantiated:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            contains:
              docker.container.name: alpha
          config:
            - type: log
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log

              multiline.pattern: '^({"log":")[0-9]{4}-[0-9]{2}-[0-9]{2}.{85}(: Mosaic response Xml)'
              multiline.negate: true
              multiline.match: after
              multiline.flush_pattern: '^({"log":",{Date=\[)[A-Za-z]{3,5},.{4}[A-Za-z]{3,5}[[:blank:]].{20}(Server)'

              fields_under_root: true
              fields:
                appid: id

        - condition:
            not.contains:
              docker.container.name: alpha
          config:
            - type: container
              paths:
                - /var/lib/docker/containers/${data.docker.container.id}/*.log

              fields_under_root: true
              fields:
                appid: id

output.file:
  path: "/var/lib/docker/filebeat"

processors:
  - add_docker_metadata: ~

Let's see if this works.

If you are going to have more different configurations, you may consider using hints-based autodiscover, that allows you to define a default configuration that can be overriden per pod or per container using annotations.

1 Like
3

Absolutely perfect! I changed the paths in both conditions to be the same and it solved the issue right away. Thank you very much for your help!

1 Like
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.