# Remote cluster monitoring creating alerts with percentage values

**URL:** <https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477>\
**Category:** Kibana\
**Tags:** elastic-stack-alerting\
**Created:** [April 7, 2021, 1:55pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477 "2021-04-07T13:55:22Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![jugonzalezv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jugonzalezv/32/85190_2.png) [@jugonzalezv](https://discuss.elastic.co/u/jugonzalezv)\
**Post date:** [April 7, 2021, 1:55pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/1 "2021-04-07T13:55:23Z")

</div>

Good day everyone, lately I've been trying to create alarms on remotely connected clusters using the following command on the clusters i connect:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/a/0a004c720880eb41a0a9de690f96f53b73c2fe6e.png)

the connection part has been succesful so far, but I'm posting it to create a better context.

the problem I've been running into is that while trying to find cpu and disk percentage values in the clusters I'm monitoring through the stack either using .monitoring\* or monitoring-es-\*

the values are not there, but if i go to stack monitoring I'm able to see said values are arriving

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/3/03dffc9544f519c4a0ea9ecc5ca570f410fb40cf.png)

my question is: Where am I able to find these values? Am I searching for these values in the wrong place?

I have tried looking for os.cpu.percentages in the stack but the only thing close to a percentage i am able to find is the used JVM heap memory

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [April 7, 2021, 4:22pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/2 "2021-04-07T16:22:28Z")

</div>

Hi @jugonzalezv ,

I can help!

> [@jugonzalezv](#):
>
> trying to find cpu and disk percentage values in the clusters

What does this mean? You mentioned. you were able to see values in Stack Monitoring but where are you unable to see these values?

---

<div class="post-metadata">

**Author:** ![jugonzalezv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jugonzalezv/32/85190_2.png) [@jugonzalezv](https://discuss.elastic.co/u/jugonzalezv)\
**Post date:** [April 7, 2021, 5:55pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/3 "2021-04-07T17:55:11Z")

</div>

Hi!

I made an Index pattern that receives all the metrics that are sent by the connected clusters, .monitoring\*, however inside this index I'm unable to find said fields that cotain the percents.

thanks for answering!

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [April 7, 2021, 6:07pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/4 "2021-04-07T18:07:51Z")

</div>

Where are you not seeing them? In Discover? Or In Metrics UI? Or are you querying against the index in Dev Tools?

---

<div class="post-metadata">

**Author:** ![jugonzalezv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jugonzalezv/32/85190_2.png) [@jugonzalezv](https://discuss.elastic.co/u/jugonzalezv)\
**Post date:** [April 7, 2021, 6:42pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/5 "2021-04-07T18:42:21Z")

</div>

I'm unable to find them whenever i do a cat mapping, or search in Discover, but as i said before the percent values do appear in the stack monitoring.

thanks again for answering!

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [April 7, 2021, 7:12pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/6 "2021-04-07T19:12:20Z")

</div>

Some fields are stored within `.monitoring-*` indices but do not exist in the mapping as they are not used for any filtering or aggregations. They just exist in the source document. I don't think Discover will show these either, as they are not mapped.

If you do a

```auto
POST .monitoring-es-*/_search
{
  "size": 1,
  "query": {
    "term": {
      "type": {
        "value": "cluster_stats"
      }
    }
  }
}

```

in Dev Tools, you can see exactly what's in the document(s)

---

<div class="post-metadata">

**Author:** ![jugonzalezv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jugonzalezv/32/85190_2.png) [@jugonzalezv](https://discuss.elastic.co/u/jugonzalezv)\
**Post date:** [April 7, 2021, 7:59pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/7 "2021-04-07T19:59:09Z")

</div>

Thanks!

I tried the command you gave me it worked to solve 1 out of 2 problems i have

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5ef7f778b56407aba86e998ff893f5810cb89097.png)

but now looking at CPU the only value I'm getting is inside a process and not inside os

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [April 8, 2021, 1:56pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/8 "2021-04-08T13:56:01Z")

</div>

I don't think we report on overall OS cpu, as these metrics are for Elastic stack products only.

If you are looking for OS level metrics, you should consider using [beats](https://www.elastic.co/beats/) to get this information. Unfortunately, it will be in a different format than the monitoring data (and therefore will not show up in the Stack Monitoring UI) but you can ingest both and create a dashboard showing any information you'd like.

---

<div class="post-metadata">

**Author:** ![jugonzalezv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jugonzalezv/32/85190_2.png) [@jugonzalezv](https://discuss.elastic.co/u/jugonzalezv)\
**Post date:** [April 8, 2021, 5:58pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/9 "2021-04-08T17:58:48Z")

</div>

sorry! I didn't mean to say that, i got confused

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/f/0f14e596c45a45b597f4b781db095eec1b4f2602.png)

I managed to find disk and memory percentages with the comand you gave me, but this CPU usage value has been giving me headaches lately as i'm unable to find it even with the instructions you gave me.

thank you so much with the support you have given me so far, I've managed to learn new things!

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [April 8, 2021, 6:30pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/10 "2021-04-08T18:30:55Z")

</div>

That is sourced from either `node_stats.process.cpu.percent` within `type: node_stats` for `.monitoring-es-*` indices, or it could be using cgroup data which is calculated by looking at data within `node_stats.os.cgroup`

---

<div class="post-metadata">

**Author:** ![jugonzalezv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jugonzalezv/32/85190_2.png) [@jugonzalezv](https://discuss.elastic.co/u/jugonzalezv)\
**Post date:** [April 8, 2021, 7:19pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/11 "2021-04-08T19:19:59Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/0/e/0e4343f2116a650b3a84cb1d0f25f919f4586d3b.png)

I have tried in `node_stats.process.cpu.percent `but it always shows a 0 so i thought it was showing a different value, as in cgroup it just shows usage in nanos.

---

<div class="post-metadata">

**Author:** ![jugonzalezv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jugonzalezv/32/85190_2.png) [@jugonzalezv](https://discuss.elastic.co/u/jugonzalezv)\
**Post date:** [April 9, 2021, 4:35am UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/12 "2021-04-09T04:35:13Z")

</div>

another question has risen, I'm trying to create alarms, the fact that said values are not aggregations doesn't allow me to compare them to a threshhold, is there anyway for me to be able to create alarms?

---

<div class="post-metadata">

**Author:** ![chrisronline](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrisronline/32/28230_2.png) [@chrisronline](https://discuss.elastic.co/u/chrisronline)\
**Post date:** [April 9, 2021, 1:51pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/13 "2021-04-09T13:51:25Z")

</div>

> [@jugonzalezv](#):
>
> as in cgroup it just shows usage in nanos.

Are you running your ES nodes within containers? If so, this is expected and you can calculate CPU usage within containers roughly by `usage / (periods * quota)`. In the UI you linked above, we calculate rates of change, as these values are counters and always increase as long as the node is online so ensure you use the derivative aggregation to get the same result as the UI.

---

<div class="post-metadata">

**Author:** ![jugonzalezv](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jugonzalezv/32/85190_2.png) [@jugonzalezv](https://discuss.elastic.co/u/jugonzalezv)\
**Post date:** [April 21, 2021, 8:46pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/14 "2021-04-21T20:46:57Z")

</div>

sorry if i havent responded lately it has been a busy couple of days, I understand now the UI Results, it was a confusion I had, now i asked earlier, I'm trying to create alarms but as the fs field is not an aggregation is it possible to use it as a threshold to create and monitor my disk usage?

thanks again for the support you have given me thus far!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 19, 2021, 8:47pm UTC](https://discuss.elastic.co/t/remote-cluster-monitoring-creating-alerts-with-percentage-values/269477/15 "2021-05-19T20:47:52Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
