How to monitor indices (doc count, size...)

I'm trying to design a monitoring "architecture" (i.e. workaround) to monitor ES indices. The information of the indices themselves is being ingested by using a data stream, and here it comes why I want to monitor those indices: They usually receive peaks of documents which heavily exceed the avg ingest rate, e.g. let's say normally it receives 100 docs every minute and every now and then, 30k peaks are ingested (I have no control over that flow).

I want to monitor those indices so I can send, for example, an email using watcher, when these peaks occur. My original idea was to monitor index size, but since that info is no available anywhere other than the cluster state, I gave it up. My current idea is to use transforms to make a doc count per index, use that info in a ML job (using max detector) and alert based on the anomalies detected by the job itself.

This seems to be a reasonable solution, at least on paper, but considering that there are near 2k indices, it load the cluster heavily.

Does anyone have a better solution to achieve the same kind of monitoring?

Thanks in advance!

Welcome to our community! :smiley:

Have you looked at the (free) Monitoring that is included in the stack? Monitor a cluster | Elasticsearch Guide [8.1] | Elastic

I took a look but din't find anything that can satisfy exactly what I'm looking for. Maybe I missed something...

Elasticsearch Monitoring Metrics | Kibana Guide [8.1] | Elastic maybe. There is metrics around indexed events per second that you can setup alerts on.

It seems like a great way to manually check those metrics (index rate and so on) per index, which is actually pretty cool! However, I don't know how can that be used to alert. Is there a way to set thresholds to alert based on that metrics (Stack monitoring -> Indices -> -> Advanced)? Or maybe a way to access that information to process it with a ML job?
Thanks in advance!

There are a few ways to do that - Generating alerts for anomaly detection jobs | Machine Learning in the Elastic Stack [8.1] | Elastic, Kibana alerts | Kibana Guide [8.1] | Elastic

With Kibana Alerts I cannot find a suitable metric to alert by, as I said, I'm interested in doc count, index size, and this kind of metrics. Using ML seems like the way to go but that's why my question came in: Can I access document count, index size or similar data in some way that do not require transforms?

What my team did was to install this plugin:GitHub - prometheus-community/elasticsearch_exporter: Elasticsearch stats exporter for Prometheus
Then connected it to our existing Prometheus based monitoring infrastructure.
Now we could use Grafana to set up email alerts and have a unified monitoring environment with all our other servers/tools.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.