How to use configure filebeat to filter docker containers using hints-based autodiscovery

[Daniel_Porter]

Hi there,

I'm trying to figure out how to configure filebeat (e.g. set a condition) to harvest from certain docker containers when using hints-based autodiscover. My use case is having two instances of filebeat running, and wanting them to autodiscover different docker containers running on the same docker engine.

My non-working example is this:

filebeat.config:
   modules:
     path: ${path.config}/modules.d/*.yml
     reload.enabled: false
 
 output.elasticsearch:
   hosts: ['elasticsearch:9200']
   username: 'elastic'
   password: 'changeme'
 
 filebeat.autodiscover:
   providers:
     - type: docker
       condition:
         contains.data.container.image: elastic
       hints.enabled: true
       hints.default_config:
         type: container
         paths:
            - /var/lib/docker/containers/${data.container.id}/*.log

This doesn't filter anything; it sets harvesters for all docker containers.

I am able to disallow particular containers by setting the hint co.elastic.hints/enabled: false, but this doesn't fit my use case, as it's my intent to collect logs from all of the docker containers, I just want to control which filebeat instance does the collection.

[ChrsMark (Chris Mark)]

Hi!

You cannot use condition-based autodiscovery and at the same time hints. These are 2 different approaches. You can either use conditional templates to catch specific containers and star collecting their logs or use hints to define settings on container's annotations.

Since you want to use 2 different Filebeat instances you can define 2 different conditional templates, one for each of them. In this Filebeat A will autodiscover set A of containers and Filebeat B will discover set B. No need to use hints. In case you want to use hints to enable specific modules then I'm afraid that you need to just make your template conditions more verbose.

If you still want to use hints you can just add Drop events | Filebeat Reference [7.11] | Elastic processor to drop the events you want to exclude based on their labels for instance. This will actually do the process of the event before dropping so it's not so performant.

[Daniel_Porter]

Thanks so much for the response! Yes, the main motivation for hints was specific modules. Will use more verbose template conditions. My larger goal really distributing an ELK rig so that a variety of related applications can use it, and the hints seemed like a nice way to customize it rather than asking people to modify the config - it lets the auto discover know as little as possible. C'est la vie - generally really happy with the filebeat + metricbeat + kibana combination, made for a pretty slick internal demo last week. Thanks again.