# How to get kubernetes node labels with metricbeat? **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [February 24, 2020, 1:52pm UTC](https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667 "2020-02-24T13:52:50Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Federico\_Bevione](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/federico_bevione/32/48231_2.png) [@Federico\_Bevione](https://discuss.elastic.co/u/Federico_Bevione)\ **Post date:** [February 24, 2020, 1:52pm UTC](https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667/1 "2020-02-24T13:52:50Z")
Hi there, I'm trying to enrich `node`, or `state_node` metrics of kubernetes module with k8s node labels but without any luck. I'm using metricbeat 7.6.0. My configuration is the following for the daemonset ```auto metricbeat.modules: - module: kubernetes add_metadata: true labels.dedot: true annotations.dedot: true metricsets: - container - node - pod - system - volume host: "${HOSTNAME}" hosts: ["https://${HOSTNAME}:10250"] bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token ssl.certificate_authorities: - /etc/kubelet/ssl/kubelet.crt ``` and the following for the single-pod deployment ```auto metricbeat.modules: # State metrics from kube-state-metrics service: - module: kubernetes enabled: true labels.dedot: true annotations.dedot: true metricsets: - state_node - state_deployment - state_replicaset - state_statefulset - state_pod - state_container - state_cronjob - state_service - state_persistentvolume - state_persistentvolumeclaim period: 10s hosts: ["kube-state-metrics:8080"] add_metadata: true ``` I've read [this answer](https://discuss.elastic.co/t/metricbeat-not-processing-kubernetes-labels/144574/6), where is said that _"In 6.4 all the kubernetes metricsets will also collect labels when possible out of the box"_, so I was expecting to find them in the `node` ore `state_node` metrics. Am I missing something? Thanks ---
**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\ **Post date:** [February 24, 2020, 3:41pm UTC](https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667/2 "2020-02-24T15:41:37Z")
Hmm, could you share any logs of Metricbeat so as to check if `add_kubernetes_metadata` processor is being enabled without a problem? C. ---
**Author:** ![tobias.berg](https://avatars.discourse-cdn.com/v4/letter/t/ac8455/32.png) [@tobias.berg](https://discuss.elastic.co/u/tobias.berg)\ **Post date:** [March 2, 2020, 10:52am UTC](https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667/3 "2020-03-02T10:52:41Z")
I'm having the same issue. I have the same config as Federico and I get no kubernetes.labels.\* fields on the metricbeat posts. Here's the output from metricbeat when started. I see no reference to add\_kubernetes\_metadata: > 2020-03-02T10:40:12.993Z INFO instance/beat.go:622 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/met > ricbeat/logs] > 2020-03-02T10:40:13.016Z INFO instance/beat.go:630 Beat ID: a63ee65a-52c2-447a-9315-ac8d5ba275d4 > 2020-03-02T10:40:13.017Z INFO [api] api/server.go:62 Starting stats endpoint > 2020-03-02T10:40:13.018Z INFO [api] api/server.go:64 Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost) > 2020-03-02T10:40:13.018Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed > 2020-03-02T10:40:13.018Z INFO [beat] instance/beat.go:958 Beat info {"system\_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home" > : "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "a63ee65a-52c2-447a-9315-ac8d5ba275d4"}}} > 2020-03-02T10:40:13.018Z INFO [beat] instance/beat.go:967 Build info {"system\_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020 > -02-05T23:10:10.000Z", "version": "7.6.0"}}} > 2020-03-02T10:40:13.018Z INFO [beat] instance/beat.go:970 Go runtime info {"system\_info": {"go": {"os":"linux","arch":"amd64","max\_procs":2,"version":"go1.13.7"}}} > 2020-03-02T10:40:13.019Z INFO [beat] instance/beat.go:974 Host info > 2020-03-02T10:40:13.019Z INFO [beat] instance/beat.go:1003 Process info {"system\_info": {"process": {"capabilities": {"inheritable":["chown","dac\_override","fowner","fsetid","kill","setg > id","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"permitted":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_servi > ce","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"effective":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","au > dit\_write","setfcap"],"bounding":["chown","dac\_override","fowner","fsetid","kill","setgid","setuid","setpcap","net\_bind\_service","net\_raw","sys\_chroot","mknod","audit\_write","setfcap"],"ambient":null}, > "cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no\_new\_privs":true}, "start\_time": "2020-03-02T10:40:12 > .210Z"}}} > 2020-03-02T10:40:13.020Z INFO instance/beat.go:298 Setup Beat: metricbeat; Version: 7.6.0 > 2020-03-02T10:40:13.020Z INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'metricbeat-7.6.0' as ILM is enabled. > 2020-03-02T10:40:13.020Z INFO elasticsearch/client.go:174 Elasticsearch url: > 2020-03-02T10:40:13.021Z INFO [publisher] pipeline/module.go:110 Beat name: > 2020-03-02T10:40:13.043Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s > 2020-03-02T10:40:13.043Z INFO instance/beat.go:439 metricbeat start running. > 2020-03-02T10:40:14.587Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch()) > 2020-03-02T10:40:14.654Z INFO elasticsearch/client.go:757 Attempting to connect to Elasticsearch version 7.6.0 > 2020-03-02T10:40:14.663Z INFO [license] licenser/es\_callback.go:50 Elasticsearch license: Platinum > 2020-03-02T10:40:14.670Z INFO [index-management] idxmgmt/std.go:258 Auto ILM enable success. > 2020-03-02T10:40:14.682Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false > 2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:271 ILM policy successfully loaded. > 2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:410 Set setup.template.name to '{metricbeat-7.6.0 {now/d}-000001}' as ILM is enabled. > 2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:415 Set setup.template.pattern to 'metricbeat-7.6.0-\*' as ILM is enabled. > 2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:449 Set settings.index.lifecycle.rollover\_alias in template to {metricbeat-7.6.0 {now/d}-000001} as ILM is enabled. > 2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:453 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"m > ax\_age":"30d","max\_size":"50gb"}}}}}}} as ILM is enabled. > 2020-03-02T10:40:14.725Z INFO template/load.go:89 Template metricbeat-7.6.0 already exists and will not be overwritten. > 2020-03-02T10:40:14.725Z INFO [index-management] idxmgmt/std.go:295 Loaded index template. > 2020-03-02T10:40:14.733Z INFO [index-management] idxmgmt/std.go:306 Write alias successfully generated. > 2020-03-02T10:40:14.739Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch()) established > 2020-03-02T10:40:43.044Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":240,"time":{"ms":249}},"total > ":{"ticks":1350,"time":{"ms":1359},"value":1350},"user":{"ticks":1110,"time":{"ms":1110}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral\_id":"431dfb96-6e5c-4fd0-8d64-1 > b7f0573a445","uptime":{"ms":30094}},"memstats":{"gc\_next":29786096,"memory\_alloc":17291920,"memory\_total":167854688,"rss":97075200},"runtime":{"goroutines":94}},"libbeat":{"config":{"module":{"running": > 0}},"output":{"events":{"acked":752,"batches":21,"total":752},"read":{"bytes":25174},"type":"elasticsearch","write":{"bytes":849572}},"pipeline":{"clients":1,"events":{"active":0,"published":752,"retry" > :21,"total":752},"queue":{"acked":752}}},"metricbeat":{"kubernetes":{"state\_container":{"events":268,"success":268},"state\_deployment":{"events":63,"success":63},"state\_node":{"events":12,"success":12}, > "state\_pod":{"events":268,"success":268},"state\_replicaset":{"events":141,"success":141}}},"system":{"cpu":{"cores":2},"load":{"1":0.9,"15":0.89,"5":0.84,"norm":{"1":0.45,"15":0.445,"5":0.42}}}}}} ---
**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\ **Post date:** [March 4, 2020, 7:27am UTC](https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667/4 "2020-03-04T07:27:21Z")
Hey I would suggest using the processor standalone in `metricbeat.yml` configuration: [https://www.elastic.co/guide/en/beats/metricbeat/current/add-kubernetes-metadata.html](https://www.elastic.co/guide/en/beats/metricbeat/current/add-kubernetes-metadata.html) Also there was an issue with `kubernetes.labels.*` in `7.6` which should be fixed in `7.6.1`: [https://github.com/elastic/beats/pull/16480](https://github.com/elastic/beats/pull/16480) ---
**Author:** ![Jack\_Phan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jack_phan/32/44738_2.png) [@Jack\_Phan](https://discuss.elastic.co/u/Jack_Phan)\ **Post date:** [March 4, 2020, 8:19am UTC](https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667/5 "2020-03-04T08:19:05Z")
Any update on this? I'm having the same issue at the moment. ---
**Author:** ![Federico\_Bevione](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/federico_bevione/32/48231_2.png) [@Federico\_Bevione](https://discuss.elastic.co/u/Federico_Bevione)\ **Post date:** [March 4, 2020, 9:37am UTC](https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667/6 "2020-03-04T09:37:07Z")
Thanks for your reply. At which metricset should be added? `node`, scraped by the daemonset, or `state_node`, sc raped by the single-replica deployment? I tried both but I'm still not able to see anything. Adding the processor without any options should be enough? Furthermore, the documentation says: " The `add_kubernetes_metadata` processor annotates each event with relevant metadata **based on which Kubernetes pod the event originated from**.", what does that means? In my opinion the documentation about this processor is very superficial and confused... Looking at the modified files in the pull request that you linked I suppose the problem you are talking about is related only to the pod label, isn't it? ---
**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\ **Post date:** [March 5, 2020, 11:46am UTC](https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667/7 "2020-03-05T11:46:19Z")
Hey @Federico_Bevione! All kubernetes metricsets will add metadata by default so you don't need to add `add_metadata: true` option on your configs in oder to have this feature enabled. No need to try with `add_kubernetes_metadata` processor, since we have spot the issue. Speaking of this, I was able to reproduce it and there is a bug indeed. Thank you for reporting it! We recently added support for multiple resources in autodiscover and there were some issues like the one I mentioned in the previous post. The one we have here is another one related to the same changes. There is a PR that aims to fix this: [https://github.com/elastic/beats/pull/16834](https://github.com/elastic/beats/pull/16834) Regards, C. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 2, 2020, 11:46am UTC](https://discuss.elastic.co/t/how-to-get-kubernetes-node-labels-with-metricbeat/220667/8 "2020-04-02T11:46:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.