Packetbeat does not add kubernetes metadata

[Tim_Stoop]

Hi,

I've started a minikube (using Kubernetes 1.18.3) to test out ECK and specifically packetbeat. The minikube profile is called "packetbeat" (important, as that's the hostname for the Virtualbox VM as well) and I followed the ECK quickstart to get it up and running. Elasticsearch (single node) and Kibana are running fine and packetbeat is gathering flows as well, however, I'm unable to make it add the Kubernetes metadata to the fields.

I'm working in the default namespace and created a ClusterRoleBinding to view for the default ServiceAccount in the namespace. This is working well, if I do not do that, packetbeat will report it is unable to list the Pods on the API server.

This is the Beat config I'm using to make ECK deploy packetbeat:

    apiVersion: beat.k8s.elastic.co/v1beta1
    kind: Beat
    metadata:
      name: packetbeat
    spec:
      type: packetbeat
      version: 7.9.0
      elasticsearchRef:
        name: quickstart
      kibanaRef:
        name: kibana
      config:
        packetbeat.interfaces.device: any
        packetbeat.protocols:
        - type: http
          ports: [80, 8000, 8080, 9200]
        - type: tls
          ports: [443]
        packetbeat.flows:
          timeout: 30s
          period: 10s
        processors:
        - add_kubernetes_metadata:
            host: packetbeat
            in_cluster: true
      daemonSet:
        podTemplate:
          spec:
            terminationGracePeriodSeconds: 30
            hostNetwork: true
            automountServiceAccountToken: true # some older Beat versions are depending on this settings presence in k8s context
            dnsPolicy: ClusterFirstWithHostNet
            containers:
            - name: packetbeat
              securityContext:
                runAsUser: 0
                capabilities:
                  add:
                  - NET_ADMIN

(This is mostly a slightly modified example from the ECK example page.) However, this is not working at all. I tried it with "add_kubernetes_metadata: {}" first, but that will error with the message:

2020-08-19T14:23:38.550Z ERROR [kubernetes] kubernetes/util.go:117 kubernetes: Querying for pod failed with error: pods "packetbeat" not found {"libbeat.processor": "add_kubernetes_metadata"}

This message goes away when I add the "host: packetbeat". I'm no longer getting an error now, but I'm not getting the Kubernetes metadata either. I'm mostly interested in the namespace tag, but I'm not getting any. I do not see any additional errors in the log and it just reports monitoring details every 30 seconds at the moment.

What am I doing wrong? Any more information I can provide to help me debug this?

[Tim_Stoop]

Solved it. It's actually in the docs, but you need to read it really carefully. The Indexers and Matchers are required, although the options seem to indicate that there are "defaults", those do not work or there are none. Adding the indexer and matcher as in the docs themselves (although they've been commented out there) adds the data to the streams.