# Datastreams losing indices
**URL:** https://discuss.elastic.co/t/datastreams-losing-indices/369404
**Category:** Elastic Search
**Created:** [October 25, 2024, 8:11am UTC](https://discuss.elastic.co/t/datastreams-losing-indices/369404 "2024-10-25T08:11:17Z")
**Posts on this page:** 5
**Page:** 1
### Author:  [@sbocquet](https://discuss.elastic.co/u/sbocquet)
#### Post date: [October 25, 2024, 8:11am UTC](https://discuss.elastic.co/t/datastreams-losing-indices/369404/1 "2024-10-25T08:11:17Z")
Hi,
I'm facing some strange things with my ELK stack.
Since 2/3 days, some datastreams have lost some indices, storage size went to 0 docs and 0b ! Those indices are random in the timeline.
It also seems to not write anymore in new indices...
Any idea what could go wrong ?
P.S.: At the moment, the elastic disk storage is full at 90% (100Go left) on the hot node, and 85% (73Go left) on the warm node.
---
### Author:  [@Artem\_Shelkovnikov](https://discuss.elastic.co/u/Artem_Shelkovnikov)
#### Post date: [October 25, 2024, 8:29am UTC](https://discuss.elastic.co/t/datastreams-losing-indices/369404/2 "2024-10-25T08:29:32Z")
I would start from looking into logs for Elasticsearch - it's often pretty verbose, especially in DEBUG log level.
Can you look at the logs and maybe grep by the index names to see what Elasticsearch reported?
---
### Author:  [@sbocquet](https://discuss.elastic.co/u/sbocquet)
#### Post date: [October 25, 2024, 9:34am UTC](https://discuss.elastic.co/t/datastreams-losing-indices/369404/3 "2024-10-25T09:34:02Z")
After investigating, it seems that when you reach the 90% watermark, elastic try to free space by moving some indices to another 'same' node (hot/warm/cold), and stop writing/indexing new entries.
But in my case, I only have 1 hot and 1 warm node, without replicas.
I managed to delete some old indices to gain free space, and elastic start rebalancing the indices.
To solve this with a more long term solution, I guess I have to increase the size of my data disks.
Is it possible to change those watermarks to 95% for example, because I loose near 100Go of disk space on the hot node with this one ?
---
### Author:  [@Artem\_Shelkovnikov](https://discuss.elastic.co/u/Artem_Shelkovnikov)
#### Post date: [October 25, 2024, 9:47am UTC](https://discuss.elastic.co/t/datastreams-losing-indices/369404/4 "2024-10-25T09:47:12Z")
You can indeed update watermarks following the docs: [Fix watermark errors | Elasticsearch Guide [8.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/fix-watermark-errors.html)
This should be the setting you're looking for: [Cluster-level shard allocation and routing settings | Elasticsearch Guide [8.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-cluster.html#cluster-routing-watermark-high)
> `cluster.routing.allocation.disk.watermark.high` [](https://www.elastic.co/cloud/elasticsearch-service/signup?page=docs&placement=docs-body)
> ([Dynamic](https://www.elastic.co/guide/en/elasticsearch/reference/current/settings.html#dynamic-cluster-setting)) Controls the high watermark. It defaults to `90%`, meaning that Elasticsearch will attempt to relocate shards away from a node whose disk usage is above 90%. It can alternatively be set to a ratio value, e.g., `0.9`. It can also be set to an absolute byte value (similarly to the low watermark) to relocate shards away from a node if it has less than the specified amount of free space. This setting affects the allocation of all shards, whether previously allocated or not.
```auto
PUT _cluster/settings
{
"persistent": {
"cluster.routing.allocation.disk.watermark.high": "95%"
}
}
```
However, it's still highly recommended to increase number of nodes or increase the storage size - at some point you'll reach this watermark and there will be no way out of it.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 22, 2024, 9:47am UTC](https://discuss.elastic.co/t/datastreams-losing-indices/369404/5 "2024-11-22T09:47:31Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.