Docker autodiscover seems to ignore close_inactive option


(Lars Levie) #1

I've got the following configuration:

filebeat.autodiscover:
  providers:
    - type: docker
      container.ids:
        - "*"
      hints.enabled: true
      close_inactive: 7m
      ignore_older: 7d
      processors:
        - add_docker_metadata: ~

I've set close_inactive: 7m but I am still seeing Filebeat log messages like Closing because close_inactive of 5m0s reached.

Does Docker autodiscover not support the close_* options?


(Steffen Siering) #2

The settings close_inactive, ignore_older, container.ids, and processors are not part of the providers settings. You must use templates or appenders in order to modify the generated configuration.

Are you sure you need auto-discovery? I see you have enabled hints, but then the container.ids setting suggests you just want to collect all plain docker logs, without any further specialised processing. Just configuring the docker input will get you all container log files:

filebeat.inputs:
- type: docker
  container.ids:
    - "*"
  close_inactive: 7m
  ignore_older: 7d
  processors:
    - add_docker_metadata: ~

When using the docker auto discovery module, you don't need add_docker_metadata. The containers meta data should be available. Plus, auto-discovery configures one input per found file. That is, it sets container.ids for each input to the container started. Setting container.ids to '*' in auto discovery might configure filebeat to try to create another set of harvesters for all logs every time a container is started.

If you indeed want to use docker auto-discovery with hints support, using appenders should look like this (Note: I haven't confirmed the config):

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true
      appenders:
      - type: config
        config:
          close_inactive: 7m
          ignore_older: 7d

(Lars Levie) #3

Whether I need autodiscovery or not is an excellent question. My understanding of the docs is that autodiscovery is the only way I will get logs from containers that start after Filebeat starts up, e.g. for one-off docker run commands. Is that a correct interpretation?

You're correct that I want all container logs, so I don't need hints.

When I try the config appender in your example, I get an error saying that it is not available (Exiting: error in autodiscover provider settings: unknown autodiscover appender config). It looks like there is an open PR to fix this: https://github.com/elastic/beats/pull/7846. Once that is released, I'll give it another try.

Thanks for your help!


(Steffen Siering) #4

When configuring the inputs without autodiscovery a scanner will check the for new log files every now and then. One can start and collect logs for containers, even after filebeat is started up. As filebeat actively scans for new files, logs of very short lived containers might be lost, if the container is deleted immediately. But even with auto-discovery one must hope for some good timing on very very short lived containers. Collecting logs via external processes is always subject to races.

The value auto-discovery adds is (just some coming to mind):

  • get meta-data information when container is started (no need for more expensive lookup)
  • input can be started more timely, as input type is created when docker sends new container start/stop events
  • customize log collection (e.g. multiline) based on container meta-data
  • configure filebeat module (uses ingest node for parsing) based on container type (e.g. parse nginx or apache2 logs)
  • with hints enabled users can configure multiline and other log collection features via docker labels attached to their containers

(Lars Levie) #5

Thank you for the extra details on the benefits of autodiscover. I'm dropping it in favor of this config:

filebeat.inputs:
  - type: docker
    containers.ids: "*"
    close_inactive: 7m
    ignore_older: 168h
    processors:
      - add_docker_metadata: ~

(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.