# Kubernetes Pod and container memory usage always at 0

**URL:** <https://discuss.elastic.co/t/kubernetes-pod-and-container-memory-usage-always-at-0/317665>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [October 28, 2022, 7:32am UTC](https://discuss.elastic.co/t/kubernetes-pod-and-container-memory-usage-always-at-0/317665 "2022-10-28T07:32:36Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![sebglon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebglon/32/112637_2.png) [@sebglon](https://discuss.elastic.co/u/sebglon)\
**Post date:** [October 28, 2022, 7:32am UTC](https://discuss.elastic.co/t/kubernetes-pod-and-container-memory-usage-always-at-0/317665/1 "2022-10-28T07:32:36Z")

</div>

Hi,  
I have an issue on MetricBeat v8.4.2.  
I use it to collect Kubernetes (v1.22.2) metrics.  
But i always have `kubernetes.container.memory.usage.bytes` and `kubernetes.pod.memory.usage.bytes` at 0

Here are my clusterRoles:

```auto
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metricbeat
  labels:
    k8s-app: metricbeat
rules:
  - apiGroups: [""]
    resources:
      - nodes
      - namespaces
      - events
      - pods
      - services
    verbs: ["get", "list", "watch"]
  # Enable this rule only if planing to use Kubernetes keystore
  #- apiGroups: [""]
  # resources:
  # - secrets
  # verbs: ["get"]
  - apiGroups: ["extensions"]
    resources:
      - replicasets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["apps"]
    resources:
      - statefulsets
      - deployments
      - replicasets
    verbs: ["get", "list", "watch"]
  - apiGroups: ["batch"]
    resources:
      - jobs
      - cronjobs
    verbs: ["get", "list", "watch"]
  - apiGroups:
      - ""
    resources:
      - nodes/stats
    verbs:
      - get
  - nonResourceURLs:
      - "/metrics"
    verbs:
      - get

```

If i run this cmd under the metricbeat daemonset pod:  
`curl -k https://${NODE_NAME}:10250/stats/summary --header "Authorization: Bearer $TOKEN"`  
I have my pod memory usage:

```auto
 {
   "podRef": {
    "name": "my-pod",
    "namespace": "default",
    "uid": "7be45b82-0395-4ae9-a0ab-7c275c121565"
   },
   "startTime": "2022-10-27T15:42:12Z",
   "containers": [
    {
     "name": "postgres",
     "startTime": "2022-10-27T15:42:15Z",
     "cpu": {
      "time": "2022-10-27T16:02:59Z",
      "usageNanoCores": 3272512,
      "usageCoreNanoSeconds": 19228071000
     },
     "memory": {
      "time": "2022-10-27T16:02:59Z",
      "workingSetBytes": 148652032
     },
     "rootfs": {
      "time": "2022-10-27T16:02:52Z",
      "availableBytes": 110666403840,
      "capacityBytes": 135102586880,
      "usedBytes": 196608,
      "inodesFree": 8147061,
      "inodes": 8380416,
      "inodesUsed": 60
     },
     "logs": {
      "time": "2022-10-27T16:02:59Z",
      "availableBytes": 110666403840,
      "capacityBytes": 135102586880,
      "usedBytes": 45056,
      "inodesFree": 8147061,
      "inodes": 8380416,
      "inodesUsed": 1
     }
    }
   ],
   "cpu": {
    "time": "2022-10-27T16:02:58Z",
    "usageNanoCores": 3221979,
    "usageCoreNanoSeconds": 19417139000
   },
   "memory": {
    "time": "2022-10-27T16:02:58Z",
    "availableBytes": 7367151616,
    "usageBytes": 190943232,
    "workingSetBytes": 149041152,
    "rssBytes": 41623552,
    "pageFaults": 803187,
    "majorPageFaults": 0
   },

```

After small investigation container memory and CPU usage are removed from kubelet [here](https://github.com/kubernetes/kubernetes/blob/release-1.22/pkg/kubelet/metrics/collectors/resource_metrics.go)

But on metricbeat, we calculate the pod memory usage based on the container memory usage [here](https://github.com/elastic/beats/blob/55037619954006cd2d5978081d7d6496c2c5c3b3/metricbeat/module/kubernetes/pod/data.go#L63).  
That's an issue for me to calculate the pod.memory.usageBytes with a non existing metrics. we have to reuse the provided values from kubelet

Related issue: [We are not seeing metrics for cpu, memory and network · Issue #31124 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/31124)

---

<div class="post-metadata">

**Author:** ![Giuseppe\_Santoro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/giuseppe_santoro/32/104302_2.png) [@Giuseppe\_Santoro](https://discuss.elastic.co/u/Giuseppe_Santoro)\
**Post date:** [November 4, 2022, 3:01pm UTC](https://discuss.elastic.co/t/kubernetes-pod-and-container-memory-usage-always-at-0/317665/2 "2022-11-04T15:01:19Z")

</div>

Hello @sebglon ,  
I'm here to help with your issue.

Do you mind providing some more info?

- are you running on a cloud provider?
- can you provide the entire manifest that you are using to run metricbeat?

I just tried to replicate the issue running locally with a Kind cluster with v1.22.2 and I can correctly see all the cpu and memory metrics from Kubelet

(I have reducted some parts of the stats/summary response just to keep the doc small.

```auto
{
    "node": {
        "nodeName": "multi-v1.22.2-worker2",
        "systemContainers": [
            {
                "name": "kubelet",
                "startTime": "2022-11-04T11:35:05Z",
                "cpu": {
                    "time": "2022-11-04T14:46:19Z",
                    "usageNanoCores": 10418606,
                    "usageCoreNanoSeconds": 99418972000
                },
                "memory": {
                    "time": "2022-11-04T14:46:19Z",
                    "usageBytes": 48238592,
                    "workingSetBytes": 47833088,
                    "rssBytes": 44470272,
                    "pageFaults": 95931,
                    "majorPageFaults": 0
                }
            }
        ],
        "startTime": "2022-11-03T13:41:28Z",
        "cpu": {
            "time": "2022-11-04T14:46:16Z",
            "usageNanoCores": 32453576,
            "usageCoreNanoSeconds": 287037524000
        },
        "memory": {
            "time": "2022-11-04T14:46:16Z",
            "availableBytes": 13316526080,
            "usageBytes": 424001536,
            "workingSetBytes": 266895360,
            "rssBytes": 148414464,
            "pageFaults": 3386811,
            "majorPageFaults": 37
        }
    },
    "pods": [
        {
            "podRef": {
                "name": "metricbeat-runner-j452h",
                "namespace": "kube-system",
                "uid": "719592ed-ff36-4bda-86f7-ebfb8f9104a7"
            },
            "startTime": "2022-11-04T11:39:55Z",
            "containers": [
                {
                    "name": "metricbeat",
                    "startTime": "2022-11-04T11:39:56Z",
                    "cpu": {
                        "time": "2022-11-04T14:46:18Z",
                        "usageNanoCores": 1603302,
                        "usageCoreNanoSeconds": 17067928000
                    },
                    "memory": {
                        "time": "2022-11-04T14:46:18Z",
                        "availableBytes": 165335040,
                        "usageBytes": 48029696,
                        "workingSetBytes": 44380160,
                        "rssBytes": 32710656,
                        "pageFaults": 70554,
                        "majorPageFaults": 0
                    }
                }
            ],
            "cpu": {
                "time": "2022-11-04T14:46:04Z",
                "usageNanoCores": 2155980,
                "usageCoreNanoSeconds": 17062689000
            },
            "memory": {
                "time": "2022-11-04T14:46:04Z",
                "availableBytes": 165560320,
                "usageBytes": 47804416,
                "workingSetBytes": 44154880,
                "rssBytes": 32034816,
                "pageFaults": 71478,
                "majorPageFaults": 0
            }
        }
    ]
}

```

Elastic (v8.4.2) with the Kubernetes module is correctly reporting those metrics.

So please provide more info so that I can replicate the issue. Thanks

---

<div class="post-metadata">

**Author:** ![Giuseppe\_Santoro](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/giuseppe_santoro/32/104302_2.png) [@Giuseppe\_Santoro](https://discuss.elastic.co/u/Giuseppe_Santoro)\
**Post date:** [November 4, 2022, 4:16pm UTC](https://discuss.elastic.co/t/kubernetes-pod-and-container-memory-usage-always-at-0/317665/3 "2022-11-04T16:16:41Z")

</div>

Also do you mind providing the output of the following command

```bash
15:40:19 › kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
multi-v1.22.2-control-plane Ready control-plane,master 4h7m v1.22.2 172.19.0.4 <none> Ubuntu 21.10 5.10.124-linuxkit containerd://1.5.7-13-g9d0acfe46
multi-v1.22.2-worker Ready <none> 4h6m v1.22.2 172.19.0.3 <none> Ubuntu 21.10 5.10.124-linuxkit containerd://1.5.7-13-g9d0acfe46
multi-v1.22.2-worker2 Ready <none> 4h6m v1.22.2 172.19.0.5 <none> Ubuntu 21.10 5.10.124-linuxkit containerd://1.5.7-13-g9d0acfe46
multi-v1.22.2-worker3 Ready <none> 4h6m v1.22.2 172.19.0.2 <none> Ubuntu 21.10 5.10.124-linuxkit containerd://1.5.7-13-g9d0acfe46

```

what I am looking after is mostly the container runtime.

Also I have found a similar issue (missing memory metrics) at [No memory metrics from kubelet stats endpoint · Issue #103366 · kubernetes/kubernetes · GitHub](https://github.com/kubernetes/kubernetes/issues/103366).

By reading that issue, we should be able to narrow it down by looking at the output of the following command. Can you please run the following command in your cluster and provide the output here? this is what I get in my cluster. more info at [No memory metrics from kubelet stats endpoint · Issue #103366 · kubernetes/kubernetes · GitHub](https://github.com/kubernetes/kubernetes/issues/103366#issuecomment-885430574)

```bash
root@multi-v1:/# cat /var/lib/kubelet/config.yaml | grep cgroup

cgroupDriver: cgroupfs
cgroupRoot: /kubelet

```

If the output of that command is `cgroupDriver: systemd` that might be the reason of the missing metrics.

---

<div class="post-metadata">

**Author:** ![sebglon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebglon/32/112637_2.png) [@sebglon](https://discuss.elastic.co/u/sebglon)\
**Post date:** [November 7, 2022, 4:24pm UTC](https://discuss.elastic.co/t/kubernetes-pod-and-container-memory-usage-always-at-0/317665/4 "2022-11-07T16:24:26Z")

</div>

> [@Giuseppe\_Santoro](#):
>
> `kubectl get nodes -o wide`

We not use cloud provider for Kubernetes.  
Here are the data from the cmd:  
`Debian GNU/Linux 11 (bullseye) 5.10.0-19-cloud-amd64 containerd://1.4.9`  
or  
`Debian GNU/Linux 11 (bullseye) 5.10.0-19-cloud-amd64 containerd://1.5.8`

And we use `systemd`

```auto
cat /var/lib/kubelet/config.yaml | grep cgroup
cgroupDriver: systemd

```

---

<div class="post-metadata">

**Author:** ![sebglon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebglon/32/112637_2.png) [@sebglon](https://discuss.elastic.co/u/sebglon)\
**Post date:** [November 7, 2022, 4:44pm UTC](https://discuss.elastic.co/t/kubernetes-pod-and-container-memory-usage-always-at-0/317665/5 "2022-11-07T16:44:22Z")

</div>

We have to ugrade our Kuberneter 1.22.2 to at least 1.22.9

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 5, 2022, 6:45pm UTC](https://discuss.elastic.co/t/kubernetes-pod-and-container-memory-usage-always-at-0/317665/6 "2022-12-05T18:45:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.