I'm trying to design a monitoring "architecture" (i.e. workaround) to monitor ES indices. The information of the indices themselves is being ingested by using a data stream, and here it comes why I want to monitor those indices: They usually receive peaks of documents which heavily exceed the avg ingest rate, e.g. let's say normally it receives 100 docs every minute and every now and then, 30k peaks are ingested (I have no control over that flow).
I want to monitor those indices so I can send, for example, an email using watcher, when these peaks occur. My original idea was to monitor index size, but since that info is no available anywhere other than the cluster state, I gave it up. My current idea is to use transforms to make a doc count per index, use that info in a ML job (using max detector) and alert based on the anomalies detected by the job itself.
This seems to be a reasonable solution, at least on paper, but considering that there are near 2k indices, it load the cluster heavily.
Does anyone have a better solution to achieve the same kind of monitoring?
It seems like a great way to manually check those metrics (index rate and so on) per index, which is actually pretty cool! However, I don't know how can that be used to alert. Is there a way to set thresholds to alert based on that metrics (Stack monitoring -> Indices -> -> Advanced)? Or maybe a way to access that information to process it with a ML job?
Thanks in advance!
With Kibana Alerts I cannot find a suitable metric to alert by, as I said, I'm interested in doc count, index size, and this kind of metrics. Using ML seems like the way to go but that's why my question came in: Can I access document count, index size or similar data in some way that do not require transforms?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
