I have found that filebeat is not cleaning kubernetes autodiscovered log entries from the registry. I verified this by running an ls against each source listed in my filebeat registry file. I found that the vast majority of the files in the registry were autodiscovered log files that no longer existed. I tried adding "clean_removed" and "close_removed" to the container config, but this did not make a difference. Here is the autodiscover part of my filebeat.yml file:
filebeat.autodiscover:
providers:
- type: kubernetes
templates:
- config:
- type: container
paths:
- "/var/lib/docker/containers/${data.kubernetes.container.id}/*.log"
processors:
- add_kubernetes_metadata:
in_cluster: true
sync_period: "10m"
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/lib/docker/containers"
clean_removed: true
close_removed: true
FYI: This is a similar question to Controlling harvester settings on kubernetes autodiscovery which was closed without an answer.