# Alerting on watermark threshold - Available API or field inside elasticsearch?

**URL:** <https://discuss.elastic.co/t/alerting-on-watermark-threshold-available-api-or-field-inside-elasticsearch/339451>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring\
**Created:** [July 27, 2023, 2:30pm UTC](https://discuss.elastic.co/t/alerting-on-watermark-threshold-available-api-or-field-inside-elasticsearch/339451 "2023-07-27T14:30:24Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [July 27, 2023, 2:30pm UTC](https://discuss.elastic.co/t/alerting-on-watermark-threshold-available-api-or-field-inside-elasticsearch/339451/1 "2023-07-27T14:30:24Z")

</div>

Hi

I'm looking into a way on how to achieve monitoring of the watermark thresholds. We would like to add alerting once the first threshold passes.  
I do not want to use fixed sizes inside the alerts, but would like to use the check mechanism inside elastic which indicates the watermark state. If we reconfigure the watermark thresholds, alerts get broken.

Our set up contains an elasticsearch & grafana for alerting. Grafana simply queries indices for alerting.

If the watermark threshold is passed, we see below message in the logs:

```auto
2023-07-27 15:44:05 {"@timestamp":"2023-07-27T13:44:05.882Z", "log.level": "INFO", "message":"low disk watermark [240gb] exceeded on [YficartHQVa8ym5DxQ_p3Q][elasticsearch-8-7-1][/usr/share/elasticsearch/data] free: 234.1gb[93.2%], replicas will not be assigned to this node", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch-8-7-1][management][T#2]","log.logger":"org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor","elasticsearch.cluster.uuid":"gcrubXYdSbO15GHVNFKxoQ","elasticsearch.node.id":"YficartHQVa8ym5DxQ_p3Q","elasticsearch.node.name":"elasticsearch-8-7-1","elasticsearch.cluster.name":"docker-cluster"}

```

I can't however seem to find any API which indicates the watermark is broken. Is there any API returning such information?

The closest APIs I found:

- [GET /\_health\_report/disk](https://www.elastic.co/guide/en/elasticsearch/reference/current/health-api.html)  
-\> It does still return green status on low threshold passed  
-\> It returns however red state for flood stage
- [GET \_cat/allocation?v=true](https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-allocation.html)  
-\> It returns the disk size, but no indication of the watermark thresholds
- metricbeat indices contain: [system.filesystem.free](https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-system.html)  
-\> We have to duplicate the thresholds configured via watermark to an alert, which is undesired

If such API exists, I could use heartbeat to do a call & parse the response. Any field inside elasticsearch stored by e.g. monitoring could be used too.  
Once the data is inside an index, I can query via elasticearch and Grafana could alert.  
Reminder: I don't want to use the fixed values in the alerting, because thresholds could change in the future. So that should be transilient.

Thanks!  
Christof

---

<div class="post-metadata">

**Author:** ![chouben](https://avatars.discourse-cdn.com/v4/letter/c/e495f1/32.png) [@chouben](https://discuss.elastic.co/u/chouben)\
**Post date:** [July 28, 2023, 12:05pm UTC](https://discuss.elastic.co/t/alerting-on-watermark-threshold-available-api-or-field-inside-elasticsearch/339451/2 "2023-07-28T12:05:41Z")

</div>

It seems my only option is

- Feeding the logfile into elasticsearch via filebeat
- Filter the logs via [a processor](https://www.elastic.co/guide/en/beats/filebeat/current/filtering-and-enhancing-data.html) to only ingest WARN & ERROR logs
- Query the index & alert

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 25, 2023, 12:06pm UTC](https://discuss.elastic.co/t/alerting-on-watermark-threshold-available-api-or-field-inside-elasticsearch/339451/3 "2023-08-25T12:06:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.