Once my VM CPU loading is up to 100%, ES agent can't feed log to ES cloud. So the detection rules can't work as well. How to solve this problem?
Looks like you are overloading the cluster, so either reduce the load or assign more resources.
The ES cloud cluster was not overloading.
Actually it's my monitoring target VM (installed with ES agent) was overloading, and I found during the timeframe of 100% CPU overload, there is no any metrics were ingested. So no detection can be triggered in this timeframe.
Maybe I elaborate not good enough before. When my target monitoring host (with ES agent installed) CPU is up to 100% load, I found the metric "host.cpu.usage" is no longer to be found in the data feeding in. So basically the CPU load detection rule can't be triggered.
I would like to see any experience for you guys to handle such situation? Thanks!
In situations like that, no-data alerts can generate alerts.
Additional resource for your reference is here
You may configure index threshold alert if the number of documents in your index is lower than some specific value for a specific time interval.
© 2020. All Rights Reserved - Elasticsearch
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant logo are trademarks of the Apache Software Foundation in the United States and/or other countries.