# Metricbeat not processing kubernetes labels

**URL:** <https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [August 15, 2018, 6:56pm UTC](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574 "2018-08-15T18:56:31Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![fbcbarbosa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbcbarbosa/32/34515_2.png) [@fbcbarbosa](https://discuss.elastic.co/u/fbcbarbosa)\
**Post date:** [August 15, 2018, 6:56pm UTC](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574/1 "2018-08-15T18:56:31Z")

</div>

Hello,

I'm running Metrict beat 6.3.1 on a Kubernetes clusters on AWS. It works fine, except for the fact that the `kubernetes.labels` field is missing. The metricbeat index has 1330 fields, but searching for "label" returns only `kubernetes.replicaset.replicas.labeled`.

I'm deploying with kubectl using the config from here: [https://github.com/elastic/beats/blob/master/deploy/kubernetes](https://github.com/elastic/beats/blob/master/deploy/kubernetes)

Below is my `metricbeat-config` ConfigMap. I have added the "add\_kubernetes\_metadata" processor which was supposed to get all labels, but does not seem to work.

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-config
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      # Mounted `metricbeat-daemonset-modules` configmap:
      path: ${path.config}/modules.d/*.yml
      # Reload module configs as they change:
      reload.enabled: false

    processors:
      - add_cloud_metadata:
      - add_kubernetes_metadata:
          in_cluster: true
          
    cloud.id: ${ELASTIC_CLOUD_ID}
    cloud.auth: ${ELASTIC_CLOUD_AUTH}

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_URL}']
      pipeline: kube
---

```

I've also tried adding include\_labels/include\_annotations under `metricbeat-daemonset-modules` ConfigMap as per [https://github.com/elastic/beats/pull/7470](https://github.com/elastic/beats/pull/7470) but to no avail 😕

Metricbeat DaemonSets are not logging any errors either, so I'm completely lost Any idea what could be the issue or where else could I look for information on this?

Edit: by the way we're running filebeat on the same cluster and it can get labels just fine.

Cheers

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 16, 2018, 12:36pm UTC](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574/2 "2018-08-16T12:36:27Z")

</div>

Hi @fbcbarbosa and welcome 🙂

Indeed labels should appear, on what events you are missing the labels? could you share the modules configuration?

---

<div class="post-metadata">

**Author:** ![fbcbarbosa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbcbarbosa/32/34515_2.png) [@fbcbarbosa](https://discuss.elastic.co/u/fbcbarbosa)\
**Post date:** [August 16, 2018, 1:10pm UTC](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574/3 "2018-08-16T13:10:29Z")

</div>

Hi @jsoriano, thanks!

> [@jsoriano](#):
>
> Indeed labels should appear, on what events you are missing the labels?

On all of them. Although the ones that are relevenat to me are `container` and `pod`.

I have the following modules configured:

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-daemonset-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  system.yml: |-
    - module: system
      period: 10s
      metricsets:
        - cpu
        - load
        - memory
        - network
        - process
        - process_summary
        #- core
        #- diskio
        #- socket
      processes: ['.*']
      process.include_top_n:
        by_cpu: 5 # include top 5 processes by CPU
        by_memory: 5 # include top 5 processes by memory

    - module: system
      period: 1m
      metricsets:
        - filesystem
        - fsstat
      processors:
      - drop_event.when.regexp:
          system.filesystem.mount_point: '^/(sys|cgroup|proc|dev|etc|host|lib)($|/)'
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      hosts: ["localhost:10255"]

```

and

```auto
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-deployment-modules
  namespace: kube-system
  labels:
    k8s-app: metricbeat
data:
  # This module requires `kube-state-metrics` up and running under `kube-system` namespace
  kubernetes.yml: |-
    - module: kubernetes
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_pod
        - state_container
        - event
      period: 10s
      hosts: ["kube-state-metrics:8080"]

```

Also, kube-state-metrics is running.

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 17, 2018, 9:07am UTC](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574/4 "2018-08-17T09:07:06Z")

</div>

One important difference between the filebeat and metricbeat deployments that are available in the repository is that metricbeat is run in the host network (`hostNetwork: true`). With host network processors and modules that need connectivity with the API server may require additional configuration. If this is the problem in your case you may be seeing errors when starting metricbeat like `kubernetes: Querying for pod failed with error...`.

The additional configuration that may be needed is the node host, it can be added with the `host` setting:

```auto
      - add_kubernetes_metadata:
          host: ${NODE_NAME}

```

`NODE_NAME` has to be injected in the metricbeat spec by adding it to the list of environment variables like this:

```auto
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName

```

By default it should be able to add labels to pods that match by ip and port (notice that this implies that your containers need to have ports defined). If you need to set additional rules to match objects, take a look to [indexers and matchers configuration](https://www.elastic.co/guide/en/beats/metricbeat/6.3/add-kubernetes-metadata.html).

---

<div class="post-metadata">

**Author:** ![fbcbarbosa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbcbarbosa/32/34515_2.png) [@fbcbarbosa](https://discuss.elastic.co/u/fbcbarbosa)\
**Post date:** [August 21, 2018, 2:58pm UTC](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574/5 "2018-08-21T14:58:27Z")

</div>

Thanks @jsoriano, but unfortunately that did not work 😕

I was indeed getting an error when starting up metricbeat like you said:

```auto
ERROR	kubernetes/util.go:73	kubernetes: Querying for pod failed with error: %!(EXTRA string=kubernetes api: Failure 404 pods "ip-10-11-13-123" not found)

```

I've added the NODE name variable as you said and the error is gone. Testing shows that the variable is loaded correctly:

```auto
$ kubectl exec -n kube-system -it metricbeat-t22pp -- env | grep NODE
NODE_NAME=ip-10-11-12-11.ec2.internal

```

And my containers do have ports defined. Also, I guess this shouldn't be the issue, as I am getting a ton of metrics, just not k8s labels/annotations (cloud metadata works fine).

Isn't there some configuration that works out of the box for a kubernetes cluster? Maybe one that does not run on hostNetwork? For filebeat I just ran the helm chart and it worked like a charm (no official helm chart is available for metricbeat though).

EDIT: I'm not sure hostNetwork would be the way to go though. According to [add\_kubernetes\_metadata](https://www.elastic.co/guide/en/beats/metricbeat/6.3/add-kubernetes-metadata.html), `host` is only required when `in_cluster` is set to `false`, which is not the case, as I am running metricbeat as part of the cluster. And if I set `in_cluster` to `false`, metricbeat breaks anyway:

```auto
2018-08-21T15:02:14.681Z	ERROR	instance/beat.go:691	Exiting: error initializing publisher: error initializing processors: `kube_config` path can't be empty when in_cluster is set to false
Exiting: error initializing publisher: error initializing processors: `kube_config` path can't be empty when in_cluster is set to false

```

---

<div class="post-metadata">

**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\
**Post date:** [August 22, 2018, 12:00pm UTC](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574/6 "2018-08-22T12:00:58Z")

</div>

Hi @fbcbarbosa,

> Isn't there some configuration that works out of the box for a kubernetes cluster? Maybe one that does not run on hostNetwork? For filebeat I just ran the helm chart and it worked like a charm (no official helm chart is available for metricbeat though).

Host network in metricbeat is recommended to collect also network metrics of the host and this is not needed in filebeat. In principle metricbeat can be run without host network mode if you are not interested on metrics of the host node.

> EDIT: I'm not sure hostNetwork would be the way to go though. According to [add\_kubernetes\_metadata](https://www.elastic.co/guide/en/beats/metricbeat/6.3/add-kubernetes-metadata.html), `host` is only required when `in_cluster` is set to `false` , which is not the case, as I am running metricbeat as part of the cluster. And if I set `in_cluster` to `false` , metricbeat breaks anyway:

Yes, the documentation is not very clear for some of these settings, we have started to do some improvements there ([#8036](https://github.com/elastic/beats/pull/8036), [#8029](https://github.com/elastic/beats/pull/8029)).

Regarding `add_kubernetes_metadata`, it is more intended to be used on events of non-kubernetes metricsets, for example to add the pod metadata to the metrics collected by other modules like nginx. This is a bit tricky when used with the `kubernetes` module, the indexers and matchers have to be modified and when it works it can unexpectedly overwrite other metrics under `kubernetes` namespace.

In 6.4 all the kubernetes metricsets will also collect labels when possible out of the box, probably the best option in your case is to wait for this release, it is expected to happen on the following days.

---

<div class="post-metadata">

**Author:** ![fbcbarbosa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fbcbarbosa/32/34515_2.png) [@fbcbarbosa](https://discuss.elastic.co/u/fbcbarbosa)\
**Post date:** [August 22, 2018, 5:46pm UTC](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574/7 "2018-08-22T17:46:18Z")

</div>

Great, I'll wait to try out the new release then.

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 19, 2018, 7:46pm UTC](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574/8 "2018-09-19T19:46:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.