Filebeat CPU utilization metrics are not normalized by default

Elastic Stack Beats
1

Can someone confirm that by using all default settings Filebeat CPU utilization metrics in Kibana stack monitoring are not normalized, e.g. for 4 CPUs the CPU utilization goes from 0% to 400%.

This question is just a first part from the original question: Kibana stack monitoring: Normalized vs not-normalized CPU utilization

2

I'm not sure I'm following.

I answered Elasticsearch CPU utilization metrics are normalized by default but assumed you were talking about cpu percentage differences between what you see in Stack Monitoring and what you see in the Infra UI (or, what you see in .monitoring-es-* documents versus what you see in metricbeat-* documents).

Did I assume correctly? Or are you talking about monitoring Filebeat within the Stack Monitoring UI?

3

what I meant is: if you enable stack monitoring in Filebeat by putting

monitoring:
  enabled: true

in filebeat.yml
are the CPU utilization metrics for Filebeat that are visible in Kibana: normalized or not normalized?

Screenshot_68
Screenshot_68.png1256×566 81.8 KB

Same thing for Elasticsearch:
If I enable metrics by adding below to elasticsearch.yml:

xpack:
  monitoring.collection.enabled: true

Are the CPU utilization metrics that are shown in Kibana: normalized or not normalized?

My conclusion was that

  • Filebeat metrics are not normalized
  • Elasticsearch metrics are normalized

I need someone to confirm that.

4

I see.

That's a good question.

I'm not sure about the exact code path, but this code leads me to believe it is not normalized across cores.

@ruflin will know more about this

5

Hi @marko_vranjkovic,

I traced through the Beats code and can confirm that the CPU usage reported to stack monitoring for a Beat is not normalized by number of cores. Details of my code trace are below.

Working backwards from the Stack Monitoring UI code to the Beats code:

  1. The chart that you see in the Stack Monitoring UI is based on the beats_stats.metrics.beat.cpu.total.value field from documents in the .monitoring-beats-* index.

  2. These documents are indexed periodically (default every 10 seconds) by the Beat.

  3. The value of this field comes from here in the Beats code.

  4. Jumping through a couple more function calls, the value of that field is taken from here. As you can see a couple lines later, Beats actually does have access to the normalized CPU percentage value, but at the moment we are not sending it to stack monitoring, so the UI cannot use it.

6

Thanks for weighing in @shaunak

@marko_vranjkovic

This is definitely a poor UX. We're going to open a set of issues to resolve this (one of which is https://github.com/elastic/kibana/issues/52889) and I'll update the issue with them.

7

@shaunak and @chrisronline
thanks a lot for your help and clarification!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.