# Docker autodiscover seems to ignore close\_inactive option **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [October 10, 2018, 9:45pm UTC](https://discuss.elastic.co/t/docker-autodiscover-seems-to-ignore-close-inactive-option/151944 "2018-10-10T21:45:54Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![larslevie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/larslevie/32/36363_2.png) [@larslevie](https://discuss.elastic.co/u/larslevie)\ **Post date:** [October 10, 2018, 9:45pm UTC](https://discuss.elastic.co/t/docker-autodiscover-seems-to-ignore-close-inactive-option/151944/1 "2018-10-10T21:45:54Z")
I've got the following configuration: ```auto filebeat.autodiscover: providers: - type: docker container.ids: - "*" hints.enabled: true close_inactive: 7m ignore_older: 7d processors: - add_docker_metadata: ~ ``` I've set `close_inactive: 7m` but I am still seeing Filebeat log messages like `Closing because close_inactive of 5m0s reached.` Does Docker autodiscover not support the `close_*` options? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [October 12, 2018, 11:02pm UTC](https://discuss.elastic.co/t/docker-autodiscover-seems-to-ignore-close-inactive-option/151944/2 "2018-10-12T23:02:18Z")
The settings `close_inactive`, `ignore_older`, `container.ids`, and `processors` are not part of the providers settings. You must use templates or [appenders](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-advanced.html#_appenders) in order to modify the generated configuration. Are you sure you need auto-discovery? I see you have enabled `hints`, but then the `container.ids` setting suggests you just want to collect all plain docker logs, without any further specialised processing. Just configuring the docker input will get you all container log files: ```auto filebeat.inputs: - type: docker container.ids: - "*" close_inactive: 7m ignore_older: 7d processors: - add_docker_metadata: ~ ``` When using the docker auto discovery module, you don't need `add_docker_metadata`. The containers meta data should be available. Plus, auto-discovery configures one input per found file. That is, it sets `container.ids` for each input to the container started. Setting `container.ids` to `'*'` in auto discovery might configure filebeat to try to create another set of harvesters for all logs every time a container is started. If you indeed want to use docker auto-discovery with [hints](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover-hints.html#configuration-autodiscover-hints) support, using appenders should look like this (Note: I haven't confirmed the config): ```auto filebeat.autodiscover: providers: - type: docker hints.enabled: true appenders: - type: config config: close_inactive: 7m ignore_older: 7d ``` ---
**Author:** ![larslevie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/larslevie/32/36363_2.png) [@larslevie](https://discuss.elastic.co/u/larslevie)\ **Post date:** [October 15, 2018, 2:41pm UTC](https://discuss.elastic.co/t/docker-autodiscover-seems-to-ignore-close-inactive-option/151944/3 "2018-10-15T14:41:21Z")
Whether I need autodiscovery or not is an excellent question. My understanding of the docs is that autodiscovery is the only way I will get logs from containers that start _after_ Filebeat starts up, e.g. for one-off `docker run` commands. Is that a correct interpretation? You're correct that I want all container logs, so I don't need `hints`. When I try the `config` appender in your example, I get an error saying that it is not available (`Exiting: error in autodiscover provider settings: unknown autodiscover appender config`). It looks like there is an open PR to fix this: [https://github.com/elastic/beats/pull/7846](https://github.com/elastic/beats/pull/7846). Once that is released, I'll give it another try. Thanks for your help! ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [October 15, 2018, 10:37pm UTC](https://discuss.elastic.co/t/docker-autodiscover-seems-to-ignore-close-inactive-option/151944/4 "2018-10-15T22:37:44Z")
When configuring the inputs without autodiscovery a scanner will check the for new log files every now and then. One can start and collect logs for containers, even after filebeat is started up. As filebeat actively scans for new files, logs of very short lived containers might be lost, if the container is deleted immediately. But even with auto-discovery one must hope for some good timing on very very short lived containers. Collecting logs via external processes is always subject to races. The value auto-discovery adds is (just some coming to mind): - get meta-data information when container is started (no need for more expensive lookup) - input can be started more timely, as input type is created when docker sends new container start/stop events - customize log collection (e.g. multiline) based on container meta-data - configure filebeat module (uses ingest node for parsing) based on container type (e.g. parse nginx or apache2 logs) - with hints enabled users can configure multiline and other log collection features via docker labels attached to their containers ---
**Author:** ![larslevie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/larslevie/32/36363_2.png) [@larslevie](https://discuss.elastic.co/u/larslevie)\ **Post date:** [October 16, 2018, 4:18pm UTC](https://discuss.elastic.co/t/docker-autodiscover-seems-to-ignore-close-inactive-option/151944/5 "2018-10-16T16:18:29Z")
Thank you for the extra details on the benefits of autodiscover. I'm dropping it in favor of this config: ```auto filebeat.inputs: - type: docker containers.ids: "*" close_inactive: 7m ignore_older: 168h processors: - add_docker_metadata: ~ ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 13, 2018, 4:24pm UTC](https://discuss.elastic.co/t/docker-autodiscover-seems-to-ignore-close-inactive-option/151944/6 "2018-11-13T16:24:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.