[solved] Kubernetes – autodiscover and add_kubermetes_metadata only working partially

[c_g (chris)]

Hi!
Basically I'm trying to ship all logs from my k8s cluster, while adding kubernetes metadata to all containers and joining some multiline output on some of them.

I only get the kubernetes: {}-data on some log lines, most do not have them.

Here's my deployment yaml, including the filebeat-config:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  filebeat.yml: |-
    filebeat.inputs:
      - type: docker
        symlinks: true
        containers:
          path: "/var/lib/docker/containers"
          ids:
            - '*'
        fields:
          type: kubernetes_log
          cluster: dev
        processors:
          - add_kubernetes_metadata:
              in_cluster: true
              namespace: ${POD_NAMESPACE}

    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition:
                or:
                  - equals:
                      kubernetes.container.name: "some-container"
                  - equals:
                      kubernetes.container.name: "some-other-container"
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  multiline.pattern: '^2'
                  multiline.negate: true
                  multiline.match: after
                  multiline.max_lines: 500
                  multiline.timeout: 5s

    processors:
      - add_cloud_metadata:

    output.kafka:
      hosts: 
        - '1.2.3.4:9092'
        - '1.2.3.5:9092'
        - '1.2.3.6:9092'
      topic: 'dev-logging'
      partition.round_robin:
        reachable_only: false
      required_acks: 1
      compression: gzip
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:6.5.3
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlogpods
          mountPath: /var/log/pods
          readOnly: true
        - name: varlogcontainers
          mountPath: /var/log/containers
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config   
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      - name: varlogcontainers
        hostPath:
          path: /var/log/containers
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1    
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system
---

Any hints are greatly appreciated

[exekias (Carlos Pérez Aradros)]

Hi @c_g,

Thank you for reporting, I see a few issues in this config:

You are mixing static docker input and autodiscover, that will create a collision between them. I would suggest to switch to use autodiscover only, that should enforce metadata to be present in all cases. You can add a template at the end, with no conditions, to act as the default setting when previous configs don't match.

Best regards

[c_g (chris)]

Hi,

thanks so much for the tip! I thought the input was necessary and autodiscover only supplements it.

So, I remove the input, and am trying to add a second template, but when I skip the condition field or leave it empty filebeat would not start, stating it needs a condition.

It does work when I negate the condition from the first block, and it seems I get my k8s metadata now, but is there a more elegant way then simply copying the condition and prepending "not:"?

Best regards, Chris

[c_g (chris)]

Updating to filebeat 7.1 solves the "empty condition" error.

So, here's my working config, for reference:

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: filebeat-config
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
data:
  filebeat.yml: |-
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          templates:
            - condition:
                or:
                  - equals:
                      kubernetes.container.name: "some-container"
                  - equals:
                      kubernetes.container.name: "some-other-container"
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  multiline.pattern: '^2'
                  multiline.negate: true
                  multiline.match: after
                  multiline.max_lines: 500
                  multiline.timeout: 5s
                  fields:
                    type: kubernetes_log
                    cluster: dev
                    multiline: "true"
            - condition:
              config:
                - type: docker
                  containers.ids:
                    - "${data.kubernetes.container.id}"
                  fields:
                    type: kubernetes_log
                    cluster: dev
                    multiline: "false"

    processors:
      - add_cloud_metadata:
      - add_kubernetes_metadata:
          in_cluster: true
          namespace: ${POD_NAMESPACE}

    output.kafka:
      hosts: 
        - '1.2.3.4:9092'
        - '1.2.3.5:9092'
        - '1.2.3.6:9092'
      topic: 'dev-logging'
      partition.round_robin:
        reachable_only: false
      required_acks: 1
      compression: gzip
---
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: filebeat
  namespace: kube-system
  labels:
    k8s-app: filebeat
    kubernetes.io/cluster-service: "true"
spec:
  template:
    metadata:
      labels:
        k8s-app: filebeat
        kubernetes.io/cluster-service: "true"
    spec:
      serviceAccountName: filebeat
      terminationGracePeriodSeconds: 30
      containers:
      - name: filebeat
        image: docker.elastic.co/beats/filebeat:7.1.0
        args: [
          "-c", "/etc/filebeat.yml",
          "-e",
        ]
        env:
        - name: POD_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        securityContext:
          runAsUser: 0
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - name: config
          mountPath: /etc/filebeat.yml
          readOnly: true
          subPath: filebeat.yml
        - name: data
          mountPath: /usr/share/filebeat/data
        - name: varlogpods
          mountPath: /var/log/pods
          readOnly: true
        - name: varlogcontainers
          mountPath: /var/log/containers
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: config
        configMap:
          defaultMode: 0600
          name: filebeat-config
      - name: varlogpods
        hostPath:
          path: /var/log/pods
      - name: varlogcontainers
        hostPath:
          path: /var/log/containers
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: data
        hostPath:
          path: /var/lib/filebeat-data
          type: DirectoryOrCreate
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: filebeat
subjects:
- kind: ServiceAccount
  name: filebeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: filebeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: filebeat
  labels:
    k8s-app: filebeat
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  verbs:
  - get
  - watch
  - list
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: filebeat
  namespace: kube-system

Logs multiline on select services, adds k8s metadata to all.