Visualization help in TSVB or Lens for CPU metrics

[julgineer (Julian K)]

Hello,

I am currently trying to build a dashboard to visualize metrics from Metricbeat.

Metricbeat is running as a DaemonSet in Kubernetes and pushes infrastructure and application metrics to Elasticsearch every minute.

Now, I want to create a dashboard with these metrics. For example, I want to visualize CPU consumption per namespace. The fields:

kubernetes.namespace, kubernetes.pod.name, kubernetes.pod.cpu.usage.nanocores

are exported by Metricbeat to achieve this.

However, I can't get the panels to display correctly.

My initial idea was to use Lens, which computes the sum of kubernetes.pod.cpu.usage.nanocores and uses kubernetes.namespace for the breakdown. This works well for the last 15 minutes, as the time bucket is set to 1 minute.

But since the time buckets are dynamically adjusted based on the selected time range, the values change when, for example, I select the last 7 days. Now the time buckets are 10 minutes, causing the values to be of course roughly 10x higher than before.

What I think is missing is calculating the average CPU usage per kubernetes.pod.name before summing it for each time bucket.

As an alternative, I tried using TSVB. I fixed the time steps, so summing kubernetes.cpu.usage.nanocores is enough. However, the issue I run into here is that if the displayed time range exceeds 24 hours, the query fetches too much data and fails.

I don't think this is a very unique use case, but I just can't seem to find a solution. Can anyone help with how I can build this kind of visualization?

Thanks in advance.

[stephenb (Stephen Brown)]

Hi @julgineer Welcome to the community.

How are you ingesting the metrics? Are you using Elastic Agent?

These metrics although I agree common use cases are not always easy to graph because of the way they are reported from K8s

I ask because there are many K8s Viz already built that perhaps you could already use or perhaps clone and adjust?

Do you have this Dashboard Loaded?

[Metrics Kubernetes] Cluster Overview

If not you can load it by just adding the Integration Under Integrations (you do not actually have to use it, just install the integrations and the Dashboards will be loaded) If you are using agent and the K8s integration it should work... if you are not using the Elastic Agent.. it still may give you directions

This vis uses

last_value(kubernetes.container.cpu.usage.nanocores)/1000000000

But you need to breakdown by the container and collapse by sum it up so all the nano cores for each container used are summed up for each bucket in the histogram. This is a "Series Sum Aggregation" in TSVB .... there are 1000 containers in the series and to get the total nano cores I need to get each one and sum them up.

Yeah it is actually a bit hard to get the percentage...

But with this Graph you get Used vs Total Cores.

Hope this helps a bit