I hope this message finds you all well. I am working in a mechanism that get the Cluster health every now and then to know if elastic is up and running (Green or Yellow) and then trigger an alert in the case the cluster becomes red.
I have noticed that this works very well in most of the situations. Although not wit the Low Watermark thresholds. for instance, I configured the "cluster.routing.allocation.disk.watermark.flood_stage" to something very low like 5%. Elastic reports correctly in the logs that a given index is set to read-only.
My question is, is there a way to get this warning programticaly via an elasticsearch API?
the _cat/nodes API seems good option when monitoring the disk consumption continously. What I am looking for is away to know if a transaction goes over the watermark. As elasticsearch reports Cluster health as green; it seems that the only way to know that the watermark has been reached is when elasticsearch's Index API returns a failed response; check if the error message is related to the watermark.
This is where having ES treat internal data like Indexes we can query would be nice, but how about pulling list of all indexes /settings and looking for read-only (there are two, see below); kinda messy and output can be large.
Or just get the water mark settings from cluster settings and then get disk used % for each node and compare; that seems easier; much less data; two API calls (settings & node stats).
Good point, in my case if the index.blocks is read only will tell me that the index cannot be written. In my environment, my indices should always be able to be written.
I think it would be good to access the elastic.logs (at least the cluster.log) through the API; this can be helpful. The admin could create easy scripts to query the logs and find Errors or Warnings.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.