Hello
I've been struggling with an issue i have with metricbeats kubernetes module. Specifically the processing of kube-state-metrics.
Versions:
kube-state-metrics 2.2.4
Metricbeat/ELK 7.15
The issue:
Scraping of kube-state-metrics endpoint should produce more metrics than it currently is. Elasticsearch dashboards expect stats on memory capacity and usage, but these aren't parsed from the kube-state-metrics endpoint. Only a small subset of data are parsed.
Expected behavior
Lines from kube-state-metrics looking like this
kube_node_status_capacity{node="raspberrypi-3",resource="memory",unit="byte"} 8.19253248e+09
kube_node_status_allocatable{node="raspberrypi-3",resource="memory",unit="byte"} 8.19253248e+09
Should be parsed into a metric looking like this
"memory": {
"allocatable": {
"bytes": 29448855552
},
"capacity": {
"bytes": 29448855552
}
},
Actual behavior
Only a few annotations/labels are sent to Elasticsearch, no memory or cpu metrics are sent. An example of what is sent is here:
"node": {
"status": {
"unschedulable": false,
"memory_pressure": "false",
"disk_pressure": "false",
"pid_pressure": "false",
"ready": "true"
},
"name": "raspberrypi-3"
},
Steps taken so far:
- Pre-installed kube-state-metrics using prometheus community helm chart (since i run arm64 and need supported version for the platform which isn't included in the metricbeat helm chart).
- Verified the endpoint exposes memory/cpu metrics.
- Specify the endpoint in the following fashion in the metricbeat helm chart
- module: kubernetes
metricsets:
- state_node
period: 10s
hosts: ["kube-state-metrics.kube-system:8080"]`
- Verified the data flows as expected to elastic.
- Debugged metricbeats deployment to see it does indeed not send memory capacity/usage data for a node to Elasticsearch. And no error is present in the log.