Index key XXX did not match any of the cached resources (intermittent issue)

[vddmpo (vdx)]

Hi,

There is intermittent problem with "add_kubernetes_metadata" (resource type: pod) when the log is not enriched. The debug logs show the following:

{"log.level":"debug","@timestamp":"2022-11-04T18:57:06.524Z","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/matchers.go","file.line":92},"message":"Incoming log.file.path value: /var/log/pods/xxx-yyy-zzz_management-c98dbbf5d-ktg6c_2198106a-d4a9-46f9-91f3-1fd068be5688/xxx-yyy-management/2.log","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-11-04T18:57:06.524Z","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/matchers.go","file.line":125},"message":"Using pod uid: 2198106a-d4a9-46f9-91f3-1fd068be5688","service.name":"filebeat","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-11-04T18:57:06.524Z","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":275},"message":"Using the following index key 2198106a-d4a9-46f9-91f3-1fd068be5688","service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2022-11-04T18:57:06.524Z","log.logger":"kubernetes","log.origin":{"file.name":"add_kubernetes_metadata/kubernetes.go","file.line":278},"message":"Index key 2198106a-d4a9-46f9-91f3-1fd068be5688 did not match any of the cached resources","service.name":"filebeat","libbeat.processor":"add_kubernetes_metadata","ecs.version":"1.6.0"}

Using kubectl get pods -n $(echo $i | cut -d":" -f1) $(echo $i | cut -d":" -f2) -o jsonpath='{.metadata.uid}' I'm able to find the pod with the UID above. In Kibana I see 90% of events with kubernetes metadata, but the rest events are not enriched.

How can I investigate deeper what is happening with kubernetes.go?

Btw, the POD mentioned above is not a new one and it is up and running some weeks already (and healthy).

Filebeat versions I tried: v8.4.1 and v8.5.0.
Kubernetes v1.24.2.
Container Runtime: containerd (1.6.8-1).

Filebeat configuration:

        filebeat:
          registry:
            flush: 10s
          inputs:
            - type: container
              scan_frequency: 1s
              close_inactive: 2h
              ingore_older: 72h
              harvester_buffer_size: 65536
              exclude_files: ['.gz$']
              exclude_lines: ["^\\s+[\\-`('.|_]"]
              paths:
                - "/var/log/pods/*/*/*"
              processors:
              - add_kubernetes_metadata:
                  host: ${NODE_NAME}
                  default_indexer.enabled: false
                  default_matchers.enabled: false
                  indexers:
                    - pod_uid:
                  matchers:
                    - logs_path:
                        logs_path: '/var/log/pods/'
                        resource_type: 'pod'