The settings close_inactive, ignore_older, container.ids, and processors are not part of the providers settings. You must use templates or appenders in order to modify the generated configuration.
Are you sure you need auto-discovery? I see you have enabled hints, but then the container.ids setting suggests you just want to collect all plain docker logs, without any further specialised processing. Just configuring the docker input will get you all container log files:
When using the docker auto discovery module, you don't need add_docker_metadata. The containers meta data should be available. Plus, auto-discovery configures one input per found file. That is, it sets container.ids for each input to the container started. Setting container.ids to '*' in auto discovery might configure filebeat to try to create another set of harvesters for all logs every time a container is started.
If you indeed want to use docker auto-discovery with hints support, using appenders should look like this (Note: I haven't confirmed the config):
Whether I need autodiscovery or not is an excellent question. My understanding of the docs is that autodiscovery is the only way I will get logs from containers that start after Filebeat starts up, e.g. for one-off docker run commands. Is that a correct interpretation?
You're correct that I want all container logs, so I don't need hints.
When I try the config appender in your example, I get an error saying that it is not available (Exiting: error in autodiscover provider settings: unknown autodiscover appender config). It looks like there is an open PR to fix this: https://github.com/elastic/beats/pull/7846. Once that is released, I'll give it another try.
When configuring the inputs without autodiscovery a scanner will check the for new log files every now and then. One can start and collect logs for containers, even after filebeat is started up. As filebeat actively scans for new files, logs of very short lived containers might be lost, if the container is deleted immediately. But even with auto-discovery one must hope for some good timing on very very short lived containers. Collecting logs via external processes is always subject to races.
The value auto-discovery adds is (just some coming to mind):
get meta-data information when container is started (no need for more expensive lookup)
input can be started more timely, as input type is created when docker sends new container start/stop events
customize log collection (e.g. multiline) based on container meta-data
configure filebeat module (uses ingest node for parsing) based on container type (e.g. parse nginx or apache2 logs)
with hints enabled users can configure multiline and other log collection features via docker labels attached to their containers
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.