Query for top xx usage elasticsearch nodes

Elastic Stack Elasticsearch
1

Trying to create a query for my cluster elasticsearch nodes.
I want it to show nodes that is more than xx% of cpu or memory or JVM heap or free space left in %

  1. Not sure if I have selected the correct field for CPU , jvm and fs.
  2. Unsure which memory field to select.
  3. Also having issue wit the EVAL. PLEASE help!!!
POST /_query?format=txt
{
  "query": """
  FROM ABCD:.monitoring-*
|   WHERE @timestamp > NOW() - 8 hours
    AND (node_stats.process.cpu.percent) > 50
    OR (node_stats.jvm.mem.heap_used_percent) > 50
|   STATS 
      process_cpu = AVG(node_stats.process.cpu.percent),
      jvm_mem = AVG(node_stats.jvm.mem.heap_used_percent),
      fs_avail = AVG(elasticsearch.node.stats.fs.total.available_in_bytes)
    BY elasticsearch.node.name
    EVAL fs_avail_gb = ROUND((TO_DOUBLE(fs_avail) / 1,073,741,824) * 100, 2)
|   WHERE process_cpu > 50
|   SORT process_cpu DESC
|   LIMIT 100
    """
}
2

Hi

You’re close the main issue is mixing raw fields and aggregated values.

A few key points:
CPU → node_stats.process.cpu.percent (correct)
JVM → node_stats.jvm.mem.heap.used_percent
Memory (OS) → node_stats.os.mem.used_percent
FS → don’t use only available space, calculate percentage used:

(available_in_bytes / total_in_bytes)

The issue with EVAL happens because you’re trying to calculate after aggregation.

Key insight: do all calculations inside STATS, not after.

3

I can't believe @Rafa_Silva you are still using AI generate your answers, instead actually writing a proper human answer. Disgusting.

The information you provided is wrong!

Anyway, @Whoami1980
from the official Elasticsearch documentation for Elastic Stack Monitoring Integration you can see these are the fields we need:

  • node_stats.process.cpu.percent
  • node_stats.jvm.mem.heap_used_percent
  • elasticsearch.node.stats.fs.total.available_in_bytes

This means your fields are correct.

I can see that "8 hours" is not valid temporal unit, you can see this from here: ES|QL time spans | Elasticsearch Reference

And you also trying to use EVAL function within STATS function. That is not allowed.. You can read about Basic ES|QL syntax | Elasticsearch Reference

Also numbers with comma are not allowed.

Below is the fixed ES|QL query. Note that this with the official ES Monitoring stack and not the built-in one (legacy and depricated), so the dataset is in metrics-elasticsearch.stack_monitoring.node_stats-default. replace as needed.

FROM metrics-elasticsearch.stack_monitoring.node_stats-default
| WHERE @timestamp >= NOW()-8h
| STATS 
      process_cpu = AVG(node_stats.process.cpu.percent),
      jvm_mem = AVG(node_stats.jvm.mem.heap_used_percent),
      fs_avail = AVG(elasticsearch.node.stats.fs.total.available_in_bytes)
    BY elasticsearch.node.name
| EVAL fs_avail = ROUND((TO_DOUBLE(fs_avail) / 1073741824) * 100, 2)
| WHERE (process_cpu > 50 OR jvm_mem > 50)
| SORT process_cpu DESC
| LIMIT 100
4

Hi!

Thank you for correcting some incorrect information I posted.

But my answers are not generated by AI.

I've even commented here at some point about the use of these mechanisms.

Thank you again for the correction.

5

@dot-mike
Thanks!!! The query did work.

However was trying to find @Rafa_Silva suggestion to use memory below but to no avail.
Memory (OS) → node_stats.os.mem.used_percent

whats the correct field for the free memory in percentage?
thanks!

6

It's not availabel because this person was using AI to answer and he was dreaming this was reality.

Take a look at Elasticsearch | Elastic integrations

here you can see 3 available paramaters:
elasticsearch.node.stats.os.cgroup.memory.control_group
elasticsearch.node.stats.os.cgroup.memory.limit.bytes
elasticsearch.node.stats.os.cgroup.memory.usage.bytes <--- probably what you want.

7

@dot-mike

Thanks. understood. now i will have to try to do get the %

Btw. i was looking at the integration link but i didnt find anything on alerts or snapshot. or i should be looking at another page. i tried but to no avail. :frowning:

official Elasticsearch documentation for Elastic Stack Monitoring Integration

8

You should be able to caluclate it with the fields I've provided in my answer :wink: I will not do the homework for you.