Metadata missing on startup

[hdost (Harold Dost)]

This question seems to be in a similar vein to a different application, but I have more than just "random"

Filebeat missing container k8 metadata

When running batch of short-lived containers (which are same docker images), on some (random) occasions k8 metadata tags are missing from some of them Also there are no errors on the Filebeat service itself when filebeat starts watching the files We have following config: filebeat.inputs: - type: docker containers.ids: - '*' processors: - add_kubernetes_metadata: ~ ... ... ... Tested with Filebeat 7.6 and with latest 7.9, same issue persists Ru…

My config is similar to theirs:

    filebeat.inputs:
    - type: container
      paths:
        - /var/log/containers/*.log
      exclude_files: [<some excludes>]
      processors:
        - add_kubernetes_metadata:
            host: ${NODE_NAME}
            in_cluster: true
            matchers:
            - logs_path:
                logs_path: "/var/log/containers/"

    output.logstash:
        hosts: ["....svc.cluster.local:<SOME_PORT>"]

I have tried a couple of variations including adding an in memory buffer, but to no avail. It seems as if the processor is not loading info into the cache before being considered ready and so no information is loaded.
This seems like almost an antipattern suggestion, but is there a way to force the processor to some sort of initial startup before processing any events. As it stands it seems as though messages are just tossed through without container or pod data, both of which are readily available.