Kubernetes module not getting resource limits

Elastic Stack Beats
1

I enabled the Kubernetes module (https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html) and everything works fine. But the kubernetes.pod.memory.usage.limit.pct field is falling back to total node allocatable memory even if there is a resource limit. Actually, all of my containers in pods have resource limits and all of the percentage numbers is based on the node maximum instead of the limit.

The percentage number and the visualization in Infra panel are becoming meaningless due to this problem. (In fact, I have been misled by the wrong percentage number for quite a long time. The docker module worked well though ) How to resolve this problem?

2

Hi @fr0der1c :slight_smile:

Can you please share more details like Metricbeat version, please?

Also, can you share the output of this commands:

kubectl get node <node name> -o jsonpath='{.status.allocatable}{"\n"}{.status.capacity}{"\n"}'
kubectl get pod -n <namespace> <pod name> -o jsonpath='{range .spec.containers[*]}{.resources}{"\n"}{end}'

We don't see anything wrong in the code that sets that value.

Thanks!

3

Sorry. I forgot to add the version information. My Metricbeat is aligned with the whole Elastic stack at 6.6.0.

# kubectl get node 10.100.1.16 -o jsonpath='{.status.allocatable}{"\n"}{.status.capacity}{"\n"}'
map[cpu:3920m ephemeral-storage:47438478872 hugepages-1Gi:0 hugepages-2Mi:0 memory:7215564Ki pods:253]
map[cpu:4 ephemeral-storage:51474044Ki hugepages-1Gi:0 hugepages-2Mi:0 memory:8167884Ki pods:253]

# kubectl get pod -n default discourse-web-b75c85f74-qb8g4 -o jsonpath='{range .spec.containers[*]}{.resources}{"\n"}{end}'
map[limits:map[cpu:1 memory:1Gi] requests:map[cpu:500m memory:500Mi]]

You can see I actually set the 1Gi memory limitation from the result above. But the pod detail says I'm using 12% of the memory.

image
image.png2760×1452 247 KB

While the container detail says I'm using more than 80%. This is a single-container pod, so the memory usage in pod detail and container detail should be the same.

image
image.png2750×1638 294 KB

4

@Mario_Castro
Hello. I've provided the information. Could you please tell me where the problem is?

5

Is this the same issue as here? https://github.com/elastic/beats/issues/10658

7

Umm... The issue link you provided references back to this thread. I'm not sure what you mean.:confused:

8

@fr0der1c Sorry, my bad. I understand why you were confused. I didn't realise that it's linking to this discuss topic. As it seems this is a bug, I suggest we continue the discussion in the Github issue.

9

@ruflin Ok. Should I provide more information there or just wait for the fix?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.