How to get kubernetes node labels with metricbeat?

Federico_Bevione · February 24, 2020, 1:52pm

Hi there,

I'm trying to enrich node, or state_node metrics of kubernetes module with k8s node labels but without any luck.
I'm using metricbeat 7.6.0.

My configuration is the following for the daemonset

metricbeat.modules:
    - module: kubernetes
      add_metadata: true
      labels.dedot: true
      annotations.dedot: true
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      host: "${HOSTNAME}"
      hosts: ["https://${HOSTNAME}:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.certificate_authorities:
        - /etc/kubelet/ssl/kubelet.crt

and the following for the single-pod deployment

metricbeat.modules:
    # State metrics from kube-state-metrics service:
    - module: kubernetes
      enabled: true
      labels.dedot: true
      annotations.dedot: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_statefulset
        - state_pod
        - state_container
        - state_cronjob
        - state_service
        - state_persistentvolume
        - state_persistentvolumeclaim
      period: 10s
      hosts: ["kube-state-metrics:8080"]
      add_metadata: true

I've read this answer, where is said that "In 6.4 all the kubernetes metricsets will also collect labels when possible out of the box", so I was expecting to find them in the node ore state_node metrics.

Am I missing something?

Thanks

ChrsMark · February 24, 2020, 3:41pm

Hmm, could you share any logs of Metricbeat so as to check if add_kubernetes_metadata processor is being enabled without a problem?

C.

tobias.berg · March 2, 2020, 10:52am

I'm having the same issue. I have the same config as Federico and I get no kubernetes.labels.* fields on the metricbeat posts.

Here's the output from metricbeat when started. I see no reference to add_kubernetes_metadata:

2020-03-02T10:40:12.993Z ricbeat/logs]
2020-03-02T10:40:13.016Z 2020-03-02T10:40:13.017Z 2020-03-02T10:40:13.018Z 2020-03-02T10:40:13.018Z 2020-03-02T10:40:13.018Z : "/usr/share/metricbeat", 2020-03-02T10:40:13.018Z -02-05T23:10:10.000Z", 2020-03-02T10:40:13.018Z 2020-03-02T10:40:13.019Z 2020-03-02T10:40:13.019Z id","setuid","setpcap"," ce","net_raw","sys_chroo dit_write","setfcap"],"b "cwd": "/usr/share/metricbeat", .210Z"}}}
2020-03-02T10:40:13.020Z 2020-03-02T10:40:13.020Z 2020-03-02T10:40:13.020Z 2020-03-02T10:40:13.021Z 2020-03-02T10:40:13.043Z 2020-03-02T10:40:13.043Z 2020-03-02T10:40:14.587Z 2020-03-02T10:40:14.654Z 2020-03-02T10:40:14.663Z 2020-03-02T10:40:14.670Z 2020-03-02T10:40:14.682Z 2020-03-02T10:40:14.682Z 2020-03-02T10:40:14.682Z 2020-03-02T10:40:14.682Z 2020-03-02T10:40:14.682Z 2020-03-02T10:40:14.682Z ax_age":"30d","max_size" 2020-03-02T10:40:14.725Z 2020-03-02T10:40:14.725Z 2020-03-02T10:40:14.733Z 2020-03-02T10:40:14.739Z 2020-03-02T10:40:43.044Z ":{"ticks":1350,"time":{ b7f0573a445","uptime":{" 0}},"output":{"events":{ :21,"total":752},"queue" "state_pod":{"events":26 INFO instance/beat.go:622 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/met
INFO instance/beat.go:630 Beat ID: a63ee65a-52c2-447a-9315-ac8d5ba275d4
INFO [api] api/server.go:62 Starting stats endpoint
INFO [api] api/server.go:64 Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)
INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
INFO [beat] instance/beat.go:958 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home"
"logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "a63ee65a-52c2-447a-9315-ac8d5ba275d4"}}}
INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020
"version": "7.6.0"}}}
INFO [beat] instance/beat.go:970 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.7"}}}
INFO [beat] instance/beat.go:974 Host info
INFO [beat] instance/beat.go:1003 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setg
net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_servi
t","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","au
ounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},
"exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-03-02T10:40:12
INFO instance/beat.go:298 Setup Beat: metricbeat; Version: 7.6.0
INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'metricbeat-7.6.0' as ILM is enabled.
INFO elasticsearch/client.go:174 Elasticsearch url:
INFO [publisher] pipeline/module.go:110 Beat name:
INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
INFO instance/beat.go:439 metricbeat start running.
INFO pipeline/output.go:95 Connecting to backoff(elasticsearch())
INFO elasticsearch/client.go:757 Attempting to connect to Elasticsearch version 7.6.0
INFO [license] licenser/es_callback.go:50 Elasticsearch license: Platinum
INFO [index-management] idxmgmt/std.go:258 Auto ILM enable success.
INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
INFO [index-management] idxmgmt/std.go:271 ILM policy successfully loaded.
INFO [index-management] idxmgmt/std.go:410 Set setup.template.name to '{metricbeat-7.6.0 {now/d}-000001}' as ILM is enabled.
INFO [index-management] idxmgmt/std.go:415 Set setup.template.pattern to 'metricbeat-7.6.0-*' as ILM is enabled.
INFO [index-management] idxmgmt/std.go:449 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.6.0 {now/d}-000001} as ILM is enabled.
INFO [index-management] idxmgmt/std.go:453 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"m
:"50gb"}}}}}}} as ILM is enabled.
INFO template/load.go:89 Template metricbeat-7.6.0 already exists and will not be overwritten.
INFO [index-management] idxmgmt/std.go:295 Loaded index template.
INFO [index-management] idxmgmt/std.go:306 Write alias successfully generated.
INFO pipeline/output.go:105 Connection to backoff(elasticsearch()) established
INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":240,"time":{"ms":249}},"total
"ms":1359},"value":1350},"user":{"ticks":1110,"time":{"ms":1110}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"431dfb96-6e5c-4fd0-8d64-1
ms":30094}},"memstats":{"gc_next":29786096,"memory_alloc":17291920,"memory_total":167854688,"rss":97075200},"runtime":{"goroutines":94}},"libbeat":{"config":{"module":{"running":
"acked":752,"batches":21,"total":752},"read":{"bytes":25174},"type":"elasticsearch","write":{"bytes":849572}},"pipeline":{"clients":1,"events":{"active":0,"published":752,"retry"
:{"acked":752}}},"metricbeat":{"kubernetes":{"state_container":{"events":268,"success":268},"state_deployment":{"events":63,"success":63},"state_node":{"events":12,"success":12},
8,"success":268},"state_replicaset":{"events":141,"success":141}}},"system":{"cpu":{"cores":2},"load":{"1":0.9,"15":0.89,"5":0.84,"norm":{"1":0.45,"15":0.445,"5":0.42}}}}}}

ChrsMark · March 4, 2020, 7:27am

Hey I would suggest using the processor standalone in metricbeat.yml configuration:

https://www.elastic.co/guide/en/beats/metricbeat/current/add-kubernetes-metadata.html

Also there was an issue with kubernetes.labels.* in 7.6 which should be fixed in 7.6.1: https://github.com/elastic/beats/pull/16480

Jack_Phan · March 4, 2020, 8:19am

Any update on this?
I'm having the same issue at the moment.

Federico_Bevione · March 4, 2020, 9:37am

Thanks for your reply.

At which metricset should be added? node, scraped by the daemonset, or state_node, sc
raped by the single-replica deployment? I tried both but I'm still not able to see anything.

Adding the processor without any options should be enough?

Furthermore, the documentation says: " The add_kubernetes_metadata processor annotates each event with relevant metadata based on which Kubernetes pod the event originated from.", what does that means?

In my opinion the documentation about this processor is very superficial and confused...

Looking at the modified files in the pull request that you linked I suppose the problem you are talking about is related only to the pod label, isn't it?

ChrsMark · March 5, 2020, 11:46am

Hey @Federico_Bevione!

All kubernetes metricsets will add metadata by default so you don't need to add add_metadata: true option on your configs in oder to have this feature enabled. No need to try with add_kubernetes_metadata processor, since we have spot the issue.

Speaking of this, I was able to reproduce it and there is a bug indeed. Thank you for reporting it! We recently added support for multiple resources in autodiscover and there were some issues like the one I mentioned in the previous post. The one we have here is another one related to the same changes.

There is a PR that aims to fix this: https://github.com/elastic/beats/pull/16834

Regards,
C.

system · April 2, 2020, 11:46am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.