Delete data older than 30 days

[lshmikanth (Venkat)]

I have an index in Elasticsearch which stores huge data. Till two weeks I was able to delete the data older than 30 days by DELETE BY QUERY with some time. At that time it's storage is 40% used. Now it has reached 77% and I am unable to delete the data more than 1000 records or 1 batch by DELETE BY QUERY. With scroll size maximum it is taking is 10000 records. I want to clear the data to avoid 100% storage usage. Please help in providing ideas to clear 30 days older data and save the rest of data. Thanks in advance

[Alex_Salgado-Elastic (Alex Salgado)]

@lshmikanth

I have an index in Elasticsearch which stores huge data. Till two weeks I was able to delete the data older than 30 days by DELETE BY QUERY with some time. At that time it's storage is 40% used. Now it has reached 77% and I am unable to delete the data more than 1000 records or 1 batch by DELETE BY QUERY. With scroll size maximum it is taking is 10000 records. I want to clear the data to avoid 100% storage usage. Please help in providing ideas to clear 30 days older data and save the rest of data. Thanks in advance

Hi @lshmikanth,

Have you considered using other methods such as:

• Index Lifecycle Management (ILM): This can be found here.
• Delete by Query with Slices: Detailed documentation is available here.

Regarding the delete by query, these articles might help you:

• Elasticsearch Delete Index Performance
• How to Delete an Index in Elasticsearch
• Elasticsearch Delete Document