Metricbeat only sending some kube-state-metrics to elastic

Elastic Stack Beats
1

Hello

I've been struggling with an issue i have with metricbeats kubernetes module. Specifically the processing of kube-state-metrics.

Versions:
kube-state-metrics 2.2.4
Metricbeat/ELK 7.15

The issue:
Scraping of kube-state-metrics endpoint should produce more metrics than it currently is. Elasticsearch dashboards expect stats on memory capacity and usage, but these aren't parsed from the kube-state-metrics endpoint. Only a small subset of data are parsed.

Expected behavior
Lines from kube-state-metrics looking like this

kube_node_status_capacity{node="raspberrypi-3",resource="memory",unit="byte"} 8.19253248e+09
kube_node_status_allocatable{node="raspberrypi-3",resource="memory",unit="byte"} 8.19253248e+09

Should be parsed into a metric looking like this

			"memory": {
				"allocatable": {
					"bytes": 29448855552
				},
				"capacity": {
					"bytes": 29448855552
				}
			},

Actual behavior
Only a few annotations/labels are sent to Elasticsearch, no memory or cpu metrics are sent. An example of what is sent is here:


    "node": {
      "status": {
        "unschedulable": false,
        "memory_pressure": "false",
        "disk_pressure": "false",
        "pid_pressure": "false",
        "ready": "true"
      },
      "name": "raspberrypi-3"
    },

Steps taken so far:

  1. Pre-installed kube-state-metrics using prometheus community helm chart (since i run arm64 and need supported version for the platform which isn't included in the metricbeat helm chart).
  2. Verified the endpoint exposes memory/cpu metrics.
  3. Specify the endpoint in the following fashion in the metricbeat helm chart
 - module: kubernetes
        metricsets:
          - state_node
        period: 10s
        hosts: ["kube-state-metrics.kube-system:8080"]`
  1. Verified the data flows as expected to elastic.
  2. Debugged metricbeats deployment to see it does indeed not send memory capacity/usage data for a node to Elasticsearch. And no error is present in the log.
2

Hi @bagdus.

Which exact version of Metricbeat are you using? Support for kube-state-metrics v2 was added in 7.15.1 and all newer versions.

1 Like
3

You're right @Michalis_Katsoulis ! I was using 7.15.0. I has completely overlooked this information.
Since i needed to patch Shell4j i needed to update anyway to 7.16.1 and now it works.
Wasted a lot of time here but still glad it was easily solved.

Thanks!

4

You are welcome @bagdus . I am glad your problem is solved!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.