Guidance setting up Kibana alarms on other cluster's metrics

Elastic Stack Kibana
, ,
#1

Hello,

We are an Elastic Service Premium subscriber. I have two Deployments setup, for non-prod and prod workloads. Both clusters are using ES 7.9.3.

I followed this documentation to setup a monitoring cluster with ES 7.11.

The non-prod and prod Deployments are writing their logs to the monitoring cluster. Also the monitoring cluster is sending its logs to itself so that daily indices are removed

On the monitoring cluster's Kibana I can go to Stack Monitoring, and I see all three clusters, I can get into each one and see metrics like:

  • CPU %
  • JVM Heap,
  • Disk data among others

Now, I want to setup some alarms on those three metrics, where the Action is sending a notification to Microsoft Teams. I went over this documentation, which seems to be the be the "new-ish" way to setup alarms from Kibana instead of Elasticsearch's Watcher

In Kibana I see some predefined alarms for the metrics I want when I go to Stack Management/Alerts and Actions:

image
image1410×1020 125 KB

But I can't (and shouldn't) change those. I tried to create my own alarms following these instructions but I don't see the CPU Usage, Disk Usage, Memory JVM Usage metrics.

Any guidance is much appreciated

#2

It looks like you are a Cloud Platinum customer? If so you can also raise this question with our Support team - https://support.elastic.co/

#3

Thanks for the suggestion @warkolm , I opened a support ticket as well.

1 Like
#4

Ok I heard back from support. I wanted to update this for future generations.

It is possible to add more actions to the default Stack Monitoring Alarms, I was just in the wrong UI. Here is the way to do it:

  • In Kibana go to Stack Monitoring
  • Click on the cluster you want to update
  • Click Enter setup mode in the top left corner
  • Click on any of the [n] alerts rectangles, drill down to the alert you care about, let's say Resource Utilization/CPU Usage
  • Click on Edit alert to change the alarm configuration, including adding an Action to post a Message to Microsoft Teams

They mentioned in the next release 7.12 you will be able to update these alarms from the Stack Management/Alerts and Actions screen