Some documented fields (e.g. kubernetes.container.image) missing from events

As per https://www.elastic.co/guide/en/beats/filebeat/current/configuration-autodiscover.html#_kubernetes, kubernetes.container.image is available in filebeat events.

Consider a deployment of filebeat on a Kubernetes cluster with the following manifest, which is the same as the manifest used in curl in https://www.elastic.co/guide/en/beats/filebeat/current/running-on-kubernetes.html#_kubernetes_deploy_manifests with just output.elasticsearch replaced with output.console:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: elastic-stack-test
  labels:
    k8s-app: filebeat
data:
  filebeat.yml: |-
    filebeat.config:
      inputs:
        # Mounted `filebeat-inputs` configmap:
        path: ${path.config}/inputs.d/*.yml
        # Reload inputs configs as they change:
        reload.enabled: false
      modules:
        path: ${path.config}/modules.d/*.yml
        # Reload module configs as they change:
        reload.enabled: false

    # To enable hints based autodiscover, remove `filebeat.config.inputs` configuration and uncomment this:
    #filebeat.autodiscover:
    #  providers:
    #    - type: kubernetes
    #      hints.enabled: true

    processors:
      - add_cloud_metadata:

    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    # output.elasticsearch:
    #   hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
    #   username: ${ELASTICSEARCH_USERNAME}
    #   password: ${ELASTICSEARCH_PASSWORD}

    output.console:
      enabled: true
      codec.json:
        pretty: true
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-inputs
  namespace: elastic-stack-test
  labels:
    k8s-app: filebeat
data:
  kubernetes.yml: |-
    - type: docker
      containers.ids:
      - "*"
      processors:
        - add_kubernetes_metadata:
            in_cluster: true
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: elastic-stack-test
  labels:
    k8s-app: filebeat
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.1.1
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: ELASTICSEARCH_HOST
          value: elasticsearch
        - name: ELASTICSEARCH_PORT
          value: "9200"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: changeme
        - name: ELASTIC_CLOUD_ID
          value:
        - name: ELASTIC_CLOUD_AUTH
          value:
        securityContext:
          runAsUser: 0
          # If using Red Hat OpenShift uncomment this:
          #privileged: true
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: inputs
          mountPath: /usr/share/filebeat/inputs.d
          readOnly: true
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: inputs
        configMap:
          defaultMode: 0600
          name: filebeat-inputs
      # data folder stores a registry of read status for all files, so we don't send everything again on a Filebeat pod restart
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: elastic-stack-test
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: elastic-stack-test
  labels:
    k8s-app: filebeat
---

Here is an event then emitted by filebeat:

{
  "@timestamp": "2019-06-11T14:21:37.797Z",
  "@metadata": {
    "beat": "filebeat",
    "type": "_doc",
    "version": "7.1.1"
  },
  "log": {
    "offset": 2671026,
    "file": {
      "path": "/var/lib/docker/containers/acdfb39605861c0b170884efc0db53107f03897b5348d930e8ebfbfa41df7479/acdfb39605861c0b170884efc0db53107f03897b5348d930e8ebfbfa41df7479-json.log"
    }
  },
  "input": {
    "type": "docker"
  },
  "kubernetes": {
    "container": {
      "name": "filebeat"
    },
    "namespace": "elastic-stack-test",
    "labels": {
      "controller-revision-hash": "1750708910",
      "k8s-app": "filebeat",
      "pod-template-generation": "1"
    },
    "pod": {
      "name": "filebeat-kngrb",
      "uid": "37ea18f7-8c54-11e9-b3d3-0683a22eb04e"
    },
    "node": {
      "name": "ip-10-10-52-185.eu-central-1.compute.internal"
    }
  },
  "cloud": {
    "region": "eu-central-1",
    "availability_zone": "eu-central-1a",
    "instance": {
      "id": "i-0e5539ca7a497da8e"
    },
    "provider": "aws",
    "machine": {
      "type": "m5.large"
    }
  },
  "host": {
    "name": "filebeat-kngrb"
  },
  "stream": "stdout",
  "message": "  },",
  "agent": {
    "hostname": "filebeat-kngrb",
    "id": "08806c22-3a7d-49e8-815b-516c9fc28463",
    "version": "7.1.1",
    "type": "filebeat",
    "ephemeral_id": "5531ccce-704e-4342-a69c-442a4b5caef1"
  },
  "ecs": {
    "version": "1.0.0"
  }
}

kubernetes.container.image (kubernetes.container.id too) is missing from the event.

I suspect this is a bug. Please confirm.

Thanks

Hi @bagratte,

These manifests don't use autodiscover by default, but a static docker input with add_kubernetes_metadata. See https://www.elastic.co/guide/en/beats/filebeat/master/add-kubernetes-metadata.html for more details.

That said, both autodiscover and add_kubernetes_metadata provide the same metadata, and container image is not part of it. Is there anything you wanted to do with it? I'm curious about this use case.

Best regards

Hi @exekias,

If kubernetes.container.image is not part of the added metadata, first of all we shall remove any mentions of it from the docs.

But yes, kubernetes.container.image is a nice piece of useful metadata. I wanted to filter the events in Logstash based on this metadata, which in my case is the cleanest way because we have a couple of deployments of the same docker image, configured differently, that produce the same format of logs. Also, in general, kubernetes.container.image is a better selector for log filtering as opposed to kubernetes.labels.some-label or kubernetes.container.name or whatnot because the format of logs is determined by it.

Regards