Filebeat is not cleaning old kubernetes log files that were autodiscovered from the registry

I have found that filebeat is not cleaning kubernetes autodiscovered log entries from the registry. I verified this by running an ls against each source listed in my filebeat registry file. I found that the vast majority of the files in the registry were autodiscovered log files that no longer existed. I tried adding "clean_removed" and "close_removed" to the container config, but this did not make a difference. Here is the autodiscover part of my filebeat.yml file:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - config:
            - type: container
              paths:
                - "/var/lib/docker/containers/${data.kubernetes.container.id}/*.log"
              processors:
                - add_kubernetes_metadata:
                    in_cluster: true
                    sync_period: "10m"
                    host: ${NODE_NAME}
                    matchers:
                        - logs_path:
                            logs_path: "/var/lib/docker/containers"
              clean_removed: true
              close_removed: true

FYI: This is a similar question to Controlling harvester settings on kubernetes autodiscovery which was closed without an answer.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.