ECK | Filebeat | Kubernetes Logs are missing / no field data

[TimK (Tim)]

Hi there!

We recently deployed the Elastic Cloud on Kubernetes for a Kubernetes Cluster in Azure. Our goal is to collect the log information from the pods.

I applied the following Filebeat YAML from the Doc (with the matching configuration for our cluster). The pods of the Filebeat run in the regarding node of the cluster without problems and also collect some information. However, under "Discover" in Kibana the available fields for Kubernetes metrics show no field data (e.g. kubernetes.name; kubernetes.namespace; kubernetes.pod.name; kubernetes.node.name ...).

I was expecting to see the same log information as in the the Kubernetes Dashboard for example. Do I miss something? Do I need to extend the configuration or make any changes to the index management?

Thank you very much and best regards.

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  type: filebeat
  version: 8.10.4
  elasticsearchRef:
    name: quickstart
  config:
    filebeat.inputs:
    - type: container
      stream: all 
      format: auto
      paths:
      - /var/log/containers/*.log
  daemonSet:
    podTemplate:
      spec:
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
          - name: varlibdockercontainers
            mountPath: /var/lib/docker/containers
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers