# Kubernetes metadata missing with add\_kubernetes\_metadata enabled

**URL:** <https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [November 4, 2020, 5:29pm UTC](https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314 "2020-11-04T17:29:21Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![xsb](https://avatars.discourse-cdn.com/v4/letter/x/a587f6/32.png) [@xsb](https://discuss.elastic.co/u/xsb)\
**Post date:** [November 4, 2020, 5:29pm UTC](https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314/1 "2020-11-04T17:29:21Z")

</div>

Hello,

I am trying to enable `add_kubernetes_metadata` to an ECK setup with no luck.

My ECK has a Beats resource that manages a filebeat DaemonSet, it's all pretty standard as I've been mostly copying the config from the official docs.

I tried to enable add\_kubernetes\_metadata with this:

```
apiVersion: beat.k8s.elastic.co/vbeta1
kind: Beat
[...]
spec:
  [...]
  config:
    filebeat.inputs:
    - type: container
      paths:
      - /var/log/containers/*.log
    processors:
    - add_kubernetes_metadata:
        matchers:
        - logs_path:
            logs_path: /var/log/continers/*.log

```

Log collection still works well, indexing and showing up in Kibana without issues. But I see no kubernetes metadata in any of the log events.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [November 5, 2020, 6:57pm UTC](https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314/2 "2020-11-05T18:57:44Z")

</div>

Hey @xsb welcome to discuss 🙂

Is this the configuration you are using? I think there is a typo there:

> [@xsb](#):
>
> ` logs_path: /var/log/continers/*.log`

The `a` in cont **a** iners is missing.

If after fixing this it still doesn't work, take a look to filebeat logs, there may be something relevant there.

---

<div class="post-metadata">

**Author:** ![xsb](https://avatars.discourse-cdn.com/v4/letter/x/a587f6/32.png) [@xsb](https://discuss.elastic.co/u/xsb)\
**Post date:** [November 6, 2020, 8:58am UTC](https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314/3 "2020-11-06T08:58:36Z")

</div>

Hello @jsoriano, thanks for replying!

I made the typo when writing the post 🤦‍♂️, the config is fine in that sense.

Filebeat logs don't show any error, and the filebeat daemonSet is running in my tests as `privileged: true` so there shouldn't be any problem obtaining the resources needed in order to work and connect to the Kubernetes API.

The Beats resource is shown as green by ECK. I am running version 7.9.2

---

<div class="post-metadata">

**Author:** ![xsb](https://avatars.discourse-cdn.com/v4/letter/x/a587f6/32.png) [@xsb](https://discuss.elastic.co/u/xsb)\
**Post date:** [November 6, 2020, 10:05am UTC](https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314/4 "2020-11-06T10:05:08Z")

</div>

When trying the Autodiscover mode (instead of add\_kubernetes\_metadata) I could see this error in the filebeat logs:

> 2020-11-06T09:26:15.041Z ERROR instance/beat.go:951 Exiting: error in autodiscover provider settings: error setting up kubernetes autodiscover provider: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES\_MASTER environment variable  
> Exiting: error in autodiscover provider settings: error setting up kubernetes autodiscover provider: unable to build kube config due to error: invalid configuration: no configuration has been provided, try setting KUBERNETES\_MASTER environment variable

I was expecting this to work as in\_cluster. It's not clear to me if there is any previous step I should do in order to make the filebeat pod know how to reach the Kubernetes API.

Also enabling DEBUG mode doesn't show any useful message.

---

<div class="post-metadata">

**Author:** ![xsb](https://avatars.discourse-cdn.com/v4/letter/x/a587f6/32.png) [@xsb](https://discuss.elastic.co/u/xsb)\
**Post date:** [November 6, 2020, 10:09am UTC](https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314/5 "2020-11-06T10:09:47Z")

</div>

I also tried to set the option `in_cluster: true` but I get the same error.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [November 6, 2020, 3:59pm UTC](https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314/6 "2020-11-06T15:59:23Z")

</div>

@xsb I have been trying and there are some extra things you need to do:

1. You need to indicate the hostname to `add_kubernetes_metadata`.
2. You need to give permissions to this pod to access the Kubernetes API.

For the first point, you need to add the `host` option to `add_kubernetes_metadata`:

```auto
    processors:
    - add_kubernetes_metadata:
        host: ${NODE_NAME}
        matchers:
        - logs_path:
            logs_path: /var/log/continers/*.log

```

`NODE_NAME` needs to be defined as an environment variable in the pod definition, like this:

```auto
        ...
        containers:
        - name: filebeat
          ...
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
        ...

```

For the second point, you need to create a cluster role and assign it to a service account:

```auto
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-beat-filebeat-quickstart
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-beat-autodiscover-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elastic-beat-autodiscover
subjects:
- kind: ServiceAccount
  name: elastic-beat-filebeat-quickstart
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-beat-autodiscover
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch

```

Then use the service account in the filebeat pod:

```auto
  ...
  daemonSet:
    podTemplate:
      spec:
        serviceAccount: elastic-beat-filebeat-quickstart
        automountServiceAccountToken: true
  ...

```

There is a section about this in the configuration docs: [https://www.elastic.co/guide/en/cloud-on-k8s/1.2/k8s-beat-configuration.html#k8s-beat-role-based-access-control-for-beats](https://www.elastic.co/guide/en/cloud-on-k8s/1.2/k8s-beat-configuration.html#k8s-beat-role-based-access-control-for-beats)

For reference, this works for me:

```auto
apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: quickstart
spec:
  type: filebeat
  version: 7.9.3
  elasticsearchRef:
    name: quickstart
  config:
    filebeat.inputs:
    - type: container
      paths:
      - /var/log/containers/*.log
    processors:
    - add_kubernetes_metadata:
       host: ${NODE_NAME}
       matchers:
       - logs_path:
           logs_path: "/var/log/containers/"
  daemonSet:
    podTemplate:
      spec:
        serviceAccount: elastic-beat-filebeat-quickstart
        automountServiceAccountToken: true
        dnsPolicy: ClusterFirstWithHostNet
        hostNetwork: true
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          env:
          - name: NODE_NAME
            valueFrom:
              fieldRef:
                fieldPath: spec.nodeName
          volumeMounts:
          - name: varlogcontainers
            mountPath: /var/log/containers
          - name: varlogpods
            mountPath: /var/log/pods
        volumes:
        - name: varlogcontainers
          hostPath:
            path: /var/log/containers
        - name: varlogpods
          hostPath:
            path: /var/log/pods
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-beat-filebeat-quickstart
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-beat-autodiscover-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: elastic-beat-autodiscover
subjects:
- kind: ServiceAccount
  name: elastic-beat-filebeat-quickstart
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-beat-autodiscover
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs:
  - get
  - list
  - watch

```

If you would like to use autodiscover, you would also need to indicate `host: ${NODE_NAME}` in the provider:

```auto
  config:  
    filebeat.autodiscover:
      providers:
        - type: kubernetes
          node: ${NODE_NAME}
          hints.enabled: true
          hints.default_config:
            type: container
            paths:
              - /var/log/containers/*${data.kubernetes.container.id}.log

```

---

<div class="post-metadata">

**Author:** ![xsb](https://avatars.discourse-cdn.com/v4/letter/x/a587f6/32.png) [@xsb](https://discuss.elastic.co/u/xsb)\
**Post date:** [November 6, 2020, 4:48pm UTC](https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314/7 "2020-11-06T16:48:12Z")

</div>

As I was testing in privileged mode, I thought I didn't need to define any RBAC. Clearly I was wrong and this fixed my issue. Thanks a lot @jsoriano!!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 4, 2020, 6:48pm UTC](https://discuss.elastic.co/t/kubernetes-metadata-missing-with-add-kubernetes-metadata-enabled/254314/8 "2020-12-04T18:48:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.