# Elasticsearch disk keep getting full when using data-stream

**URL:** <https://discuss.elastic.co/t/elasticsearch-disk-keep-getting-full-when-using-data-stream/319276>\
**Category:** Elasticsearch\
**Tags:** ilm-index-lifecycle-management, datastreams\
**Created:** [November 18, 2022, 8:49am UTC](https://discuss.elastic.co/t/elasticsearch-disk-keep-getting-full-when-using-data-stream/319276 "2022-11-18T08:49:14Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![MyNameHasDiactrics](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mynamehasdiactrics/32/113508_2.png) [@MyNameHasDiactrics](https://discuss.elastic.co/u/MyNameHasDiactrics)\
**Post date:** [November 18, 2022, 8:49am UTC](https://discuss.elastic.co/t/elasticsearch-disk-keep-getting-full-when-using-data-stream/319276/1 "2022-11-18T08:49:14Z")

</div>

This morning I woke up to this:  
 ![Screenshot 2022-11-18 at 09.34.06](https://us1.discourse-cdn.com/elastic/original/3X/5/e/5e4f6b2d7d18c17d52ee353986c74f671dd237e3.png)

This is not the first time that it happens, and when it happens it blocks everything: I cannot access Kibana, Elasticsearch doesn't answer at all, this means I also cannot delete data using the API nor scale the instance to give it more disk space.  
My only fix is to usually destroy the whole deployment and start over ...

My current set-up is the following:  
I have 2 logstash sending logs from 2 kubernetes cluster (using filebeat). The logstash output is the following:

```auto
output {
      elasticsearch {
        hosts => ["<elastic search url>"]
        user => '<user>'
        password => '<password>'
        data_stream_namespace => 'production'
      }
}    

```

They both write to a data-stream. In kibana I see that this data stream is linked to the ILM for logs. By default, this ILM keep everything forever. I change this policy to move the data to a cold storage instance after 2 days and delete it after seven days.

My issue is that this ILM doesn't seem to work / be respected. This is my cold storage instance after more than 2 days:  
 ![Screenshot 2022-11-18 at 09.42.49](https://us1.discourse-cdn.com/elastic/original/3X/2/6/26990b7cd3ba6693cad89cb3d9131021f49789f2.png)

No data seem to be written to it. In the same way, I used a ILM before were data should be deleted after 2 days and after 2 days, no data seem to have been deleted in the same way.

Therefor I have 3 questions:

1. Is there a way to debug my instance when it reach 100% disk ? I am using ElasticCloud
2. Is there a configuration to make the instance read-only when it reach 95% ?
3. I am missing something with data-stream and ILM ? Why the ILM doesn't seem to work ?

Thank you

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [November 19, 2022, 12:37am UTC](https://discuss.elastic.co/t/elasticsearch-disk-keep-getting-full-when-using-data-stream/319276/2 "2022-11-19T00:37:34Z")

</div>

Hi @MyNameHasDiactrics Welcome to the community and thank you for trying elastic cloud.

A couple things

> [@MyNameHasDiactrics](#):
>
> In kibana I see that this data stream is linked to the ILM for logs. By default, this ILM keep everything forever.

Yes so you need to update that Policy to what you want.

What do you want... you want to move from hot to cold after 1 day .. 2 Days or By Size....

How Much data are you ingesting per day? How Long do you want to keep it?

That is a very small instance how much per day / hour are you ingesting?

Lets us know and we can help with that...

---

<div class="post-metadata">

**Author:** ![MyNameHasDiactrics](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mynamehasdiactrics/32/113508_2.png) [@MyNameHasDiactrics](https://discuss.elastic.co/u/MyNameHasDiactrics)\
**Post date:** [November 23, 2022, 9:16am UTC](https://discuss.elastic.co/t/elasticsearch-disk-keep-getting-full-when-using-data-stream/319276/3 "2022-11-23T09:16:50Z")

</div>

Ok so I find out what was my issue. I misunderstood how ILM phase changes are triggered. Basically when the policy is set to delete the data after x days, it actually means (or at least that is what I understood): Delete the index x days after it was _rolled over_. By default, ILM have a policy where they rollover the index after 50Go or 30 days, which was way to high for my usage.  
I changed this and now it is working.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 21, 2022, 9:17am UTC](https://discuss.elastic.co/t/elasticsearch-disk-keep-getting-full-when-using-data-stream/319276/4 "2022-12-21T09:17:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.