Kubernetes metadata missing with add_kubernetes_metadata enabled

Elastic Stack Beats
1

Hello,

I am trying to enable add_kubernetes_metadata to an ECK setup with no luck.

My ECK has a Beats resource that manages a filebeat DaemonSet, it's all pretty standard as I've been mostly copying the config from the official docs.

I tried to enable add_kubernetes_metadata with this:

apiVersion: beat.k8s.elastic.co/vbeta1
kind: Beat
[...]
spec:
  [...]
  config:
    filebeat.inputs:
    - type: container
      paths:
      - /var/log/containers/*.log
    processors:
    - add_kubernetes_metadata:
        matchers:
        - logs_path:
            logs_path: /var/log/continers/*.log

Log collection still works well, indexing and showing up in Kibana without issues. But I see no kubernetes metadata in any of the log events.

2

Hey @xsb welcome to discuss :slight_smile:

Is this the configuration you are using? I think there is a typo there:

The a in containers is missing.

If after fixing this it still doesn't work, take a look to filebeat logs, there may be something relevant there.

3

Hello @jsoriano, thanks for replying!

I made the typo when writing the post :man_facepalming:, the config is fine in that sense.

Filebeat logs don't show any error, and the filebeat daemonSet is running in my tests as privileged: true so there shouldn't be any problem obtaining the resources needed in order to work and connect to the Kubernetes API.

The Beats resource is shown as green by ECK. I am running version 7.9.2

4

When trying the Autodiscover mode (instead of add_kubernetes_metadata) I could see this error in the filebeat logs:

2020-11-06T09:26:15.041Z ERROR instance/beat.go:951 Exiting: error in autodiscover provider settings: error setting up kubernetes autodiscover provider: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Exiting: error in autodiscover provider settings: error setting up kubernetes autodiscover provider: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

I was expecting this to work as in_cluster. It's not clear to me if there is any previous step I should do in order to make the filebeat pod know how to reach the Kubernetes API.

Also enabling DEBUG mode doesn't show any useful message.

5

I also tried to set the option in_cluster: true but I get the same error.

6

@xsb I have been trying and there are some extra things you need to do:

  1. You need to indicate the hostname to add_kubernetes_metadata.
  2. You need to give permissions to this pod to access the Kubernetes API.

For the first point, you need to add the host option to add_kubernetes_metadata:

    processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: /var/log/continers/*.log

NODE_NAME needs to be defined as an environment variable in the pod definition, like this:

        ...
        containers:
        - name: filebeat
          ...
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
        ...

For the second point, you need to create a cluster role and assign it to a service account:

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-beat-filebeat-quickstart
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-beat-autodiscover-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elastic-beat-autodiscover
subjects:
- kind: ServiceAccount
  name: elastic-beat-filebeat-quickstart
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-beat-autodiscover
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch

Then use the service account in the filebeat pod:

  ...
  daemonSet:
    podTemplate:
      spec:
        serviceAccount: elastic-beat-filebeat-quickstart
        automountServiceAccountToken: true
  ...

There is a section about this in the configuration docs: https://www.elastic.co/guide/en/cloud-on-k8s/1.2/k8s-beat-configuration.html#k8s-beat-role-based-access-control-for-beats

For reference, this works for me:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  type: filebeat
  version: 7.9.3
  elasticsearchRef:
    name: quickstart
  config:
    filebeat.inputs:
    - type: container
      paths:
      - /var/log/containers/*.log
    processors:
    - add_kubernetes_metadata:
       host: ${NODE_NAME}
       matchers:
       - logs_path:
           logs_path: "/var/log/containers/"
  daemonSet:
    podTemplate:
      spec:
        serviceAccount: elastic-beat-filebeat-quickstart
        automountServiceAccountToken: true
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-beat-filebeat-quickstart
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-beat-autodiscover-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elastic-beat-autodiscover
subjects:
- kind: ServiceAccount
  name: elastic-beat-filebeat-quickstart
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-beat-autodiscover
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch

If you would like to use autodiscover, you would also need to indicate host: ${NODE_NAME} in the provider:

  config:  
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log
1 Like
7

As I was testing in privileged mode, I thought I didn't need to define any RBAC. Clearly I was wrong and this fixed my issue. Thanks a lot @jsoriano!!

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.