Filebeat is not cleaning old kubernetes log files that were autodiscovered from the registry

[carlsoane]

I have found that filebeat is not cleaning kubernetes autodiscovered log entries from the registry. I verified this by running an ls against each source listed in my filebeat registry file. I found that the vast majority of the files in the registry were autodiscovered log files that no longer existed. I tried adding "clean_removed" and "close_removed" to the container config, but this did not make a difference. Here is the autodiscover part of my filebeat.yml file:

filebeat.autodiscover:
  providers:
    - type: kubernetes
      templates:
        - config:
            - type: container
              paths:
                - "/var/lib/docker/containers/${data.kubernetes.container.id}/*.log"
              processors:
                - add_kubernetes_metadata:
                    in_cluster: true
                    sync_period: "10m"
                    host: ${NODE_NAME}
                    matchers:
                        - logs_path:
                            logs_path: "/var/lib/docker/containers"
              clean_removed: true
              close_removed: true

FYI: This is a similar question to Controlling harvester settings on kubernetes autodiscovery which was closed without an answer.