How to get kubernetes node labels with metricbeat?

Elastic Stack Beats
1

Hi there,

I'm trying to enrich node, or state_node metrics of kubernetes module with k8s node labels but without any luck.
I'm using metricbeat 7.6.0.

My configuration is the following for the daemonset

metricbeat.modules:
    - module: kubernetes
      add_metadata: true
      labels.dedot: true
      annotations.dedot: true
      metricsets:
        - container
        - node
        - pod
        - system
        - volume
      host: "${HOSTNAME}"
      hosts: ["https://${HOSTNAME}:10250"]
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      ssl.certificate_authorities:
        - /etc/kubelet/ssl/kubelet.crt

and the following for the single-pod deployment

metricbeat.modules:
    # State metrics from kube-state-metrics service:
    - module: kubernetes
      enabled: true
      labels.dedot: true
      annotations.dedot: true
      metricsets:
        - state_node
        - state_deployment
        - state_replicaset
        - state_statefulset
        - state_pod
        - state_container
        - state_cronjob
        - state_service
        - state_persistentvolume
        - state_persistentvolumeclaim
      period: 10s
      hosts: ["kube-state-metrics:8080"]
      add_metadata: true

I've read this answer, where is said that "In 6.4 all the kubernetes metricsets will also collect labels when possible out of the box", so I was expecting to find them in the node ore state_node metrics.

Am I missing something?

Thanks

1 Like
2

Hmm, could you share any logs of Metricbeat so as to check if add_kubernetes_metadata processor is being enabled without a problem?

C.

3

I'm having the same issue. I have the same config as Federico and I get no kubernetes.labels.* fields on the metricbeat posts.

Here's the output from metricbeat when started. I see no reference to add_kubernetes_metadata:

2020-03-02T10:40:12.993Z INFO instance/beat.go:622 Home path: [/usr/share/metricbeat] Config path: [/usr/share/metricbeat] Data path: [/usr/share/metricbeat/data] Logs path: [/usr/share/met
ricbeat/logs]
2020-03-02T10:40:13.016Z INFO instance/beat.go:630 Beat ID: a63ee65a-52c2-447a-9315-ac8d5ba275d4
2020-03-02T10:40:13.017Z INFO [api] api/server.go:62 Starting stats endpoint
2020-03-02T10:40:13.018Z INFO [api] api/server.go:64 Metrics endpoint listening on: 127.0.0.1:5066 (configured: localhost)
2020-03-02T10:40:13.018Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-03-02T10:40:13.018Z INFO [beat] instance/beat.go:958 Beat info {"system_info": {"beat": {"path": {"config": "/usr/share/metricbeat", "data": "/usr/share/metricbeat/data", "home"
: "/usr/share/metricbeat", "logs": "/usr/share/metricbeat/logs"}, "type": "metricbeat", "uuid": "a63ee65a-52c2-447a-9315-ac8d5ba275d4"}}}
2020-03-02T10:40:13.018Z INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020
-02-05T23:10:10.000Z", "version": "7.6.0"}}}
2020-03-02T10:40:13.018Z INFO [beat] instance/beat.go:970 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.13.7"}}}
2020-03-02T10:40:13.019Z INFO [beat] instance/beat.go:974 Host info
2020-03-02T10:40:13.019Z INFO [beat] instance/beat.go:1003 Process info {"system_info": {"process": {"capabilities": {"inheritable":["chown","dac_override","fowner","fsetid","kill","setg
id","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"permitted":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_servi
ce","net_raw","sys_chroot","mknod","audit_write","setfcap"],"effective":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","au
dit_write","setfcap"],"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},
"cwd": "/usr/share/metricbeat", "exe": "/usr/share/metricbeat/metricbeat", "name": "metricbeat", "pid": 1, "ppid": 0, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2020-03-02T10:40:12
.210Z"}}}
2020-03-02T10:40:13.020Z INFO instance/beat.go:298 Setup Beat: metricbeat; Version: 7.6.0
2020-03-02T10:40:13.020Z INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'metricbeat-7.6.0' as ILM is enabled.
2020-03-02T10:40:13.020Z INFO elasticsearch/client.go:174 Elasticsearch url:
2020-03-02T10:40:13.021Z INFO [publisher] pipeline/module.go:110 Beat name:
2020-03-02T10:40:13.043Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-03-02T10:40:13.043Z INFO instance/beat.go:439 metricbeat start running.
2020-03-02T10:40:14.587Z INFO pipeline/output.go:95 Connecting to backoff(elasticsearch())
2020-03-02T10:40:14.654Z INFO elasticsearch/client.go:757 Attempting to connect to Elasticsearch version 7.6.0
2020-03-02T10:40:14.663Z INFO [license] licenser/es_callback.go:50 Elasticsearch license: Platinum
2020-03-02T10:40:14.670Z INFO [index-management] idxmgmt/std.go:258 Auto ILM enable success.
2020-03-02T10:40:14.682Z INFO [index-management.ilm] ilm/std.go:139 do not generate ilm policy: exists=true, overwrite=false
2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:271 ILM policy successfully loaded.
2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:410 Set setup.template.name to '{metricbeat-7.6.0 {now/d}-000001}' as ILM is enabled.
2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:415 Set setup.template.pattern to 'metricbeat-7.6.0-*' as ILM is enabled.
2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:449 Set settings.index.lifecycle.rollover_alias in template to {metricbeat-7.6.0 {now/d}-000001} as ILM is enabled.
2020-03-02T10:40:14.682Z INFO [index-management] idxmgmt/std.go:453 Set settings.index.lifecycle.name in template to {metricbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"m
ax_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2020-03-02T10:40:14.725Z INFO template/load.go:89 Template metricbeat-7.6.0 already exists and will not be overwritten.
2020-03-02T10:40:14.725Z INFO [index-management] idxmgmt/std.go:295 Loaded index template.
2020-03-02T10:40:14.733Z INFO [index-management] idxmgmt/std.go:306 Write alias successfully generated.
2020-03-02T10:40:14.739Z INFO pipeline/output.go:105 Connection to backoff(elasticsearch()) established
2020-03-02T10:40:43.044Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":240,"time":{"ms":249}},"total
":{"ticks":1350,"time":{"ms":1359},"value":1350},"user":{"ticks":1110,"time":{"ms":1110}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":16},"info":{"ephemeral_id":"431dfb96-6e5c-4fd0-8d64-1
b7f0573a445","uptime":{"ms":30094}},"memstats":{"gc_next":29786096,"memory_alloc":17291920,"memory_total":167854688,"rss":97075200},"runtime":{"goroutines":94}},"libbeat":{"config":{"module":{"running":
0}},"output":{"events":{"acked":752,"batches":21,"total":752},"read":{"bytes":25174},"type":"elasticsearch","write":{"bytes":849572}},"pipeline":{"clients":1,"events":{"active":0,"published":752,"retry"
:21,"total":752},"queue":{"acked":752}}},"metricbeat":{"kubernetes":{"state_container":{"events":268,"success":268},"state_deployment":{"events":63,"success":63},"state_node":{"events":12,"success":12},
"state_pod":{"events":268,"success":268},"state_replicaset":{"events":141,"success":141}}},"system":{"cpu":{"cores":2},"load":{"1":0.9,"15":0.89,"5":0.84,"norm":{"1":0.45,"15":0.445,"5":0.42}}}}}}

4

Hey I would suggest using the processor standalone in metricbeat.yml configuration:

https://www.elastic.co/guide/en/beats/metricbeat/current/add-kubernetes-metadata.html

Also there was an issue with kubernetes.labels.* in 7.6 which should be fixed in 7.6.1: https://github.com/elastic/beats/pull/16480

5

Any update on this?
I'm having the same issue at the moment.

6

Thanks for your reply.

At which metricset should be added? node, scraped by the daemonset, or state_node, sc
raped by the single-replica deployment? I tried both but I'm still not able to see anything.

Adding the processor without any options should be enough?

Furthermore, the documentation says: " The add_kubernetes_metadata processor annotates each event with relevant metadata based on which Kubernetes pod the event originated from.", what does that means?

In my opinion the documentation about this processor is very superficial and confused...

Looking at the modified files in the pull request that you linked I suppose the problem you are talking about is related only to the pod label, isn't it?

7

Hey @Federico_Bevione!

All kubernetes metricsets will add metadata by default so you don't need to add add_metadata: true option on your configs in oder to have this feature enabled. No need to try with add_kubernetes_metadata processor, since we have spot the issue.

Speaking of this, I was able to reproduce it and there is a bug indeed. Thank you for reporting it! We recently added support for multiple resources in autodiscover and there were some issues like the one I mentioned in the previous post. The one we have here is another one related to the same changes.

There is a PR that aims to fix this: https://github.com/elastic/beats/pull/16834

Regards,
C.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.