# Automate Deletion of Docs from an Index that are older than 10 days **URL:** \ **Category:** Elasticsearch\ **Tags:** ilm-index-lifecycle-management\ **Created:** [January 9, 2022, 1:44pm UTC](https://discuss.elastic.co/t/automate-deletion-of-docs-from-an-index-that-are-older-than-10-days/293826 "2022-01-09T13:44:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Divyank\_Mahalle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/divyank_mahalle/32/91240_2.png) [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)\ **Post date:** [January 9, 2022, 1:44pm UTC](https://discuss.elastic.co/t/automate-deletion-of-docs-from-an-index-that-are-older-than-10-days/293826/1 "2022-01-09T13:44:41Z")
Hi, Currently, I am deleting docs using DSL queries that are older than 10 days. I am having a 1P shard on a single node with No replicas or nodes for this env. DSL query- ```auto POST index_name/_delete_by_query { "query": { "range": { "@timestamp": { "lte": "now-10d" } } } } ``` As per my understanding of ILM policy, an index is getting deleted instead of docs in an index, and a new index is created. As I am putting ILM policy in newly created index, I need to confirm below configurations to work. Logs are getting ingested from logstash.I guess the output filter config needs to be changed as below. ```auto output { elasticsearch { host => <> ilm_rollover_alias => "index_name" ilm_pattern => "000001" ilm_policy => "new_policy" } } ``` Setting ILM policy Step1- ```auto PUT _ilm/policy/new_policy { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_size": "40gb" }, "set_priority": { "priority": 100 } } }, "delete": { "min_age": "10d", "actions": { "delete": {} } } } } } } ``` 2 nd step creating template- ```auto PUT _template/new_index_template { "index_patterns": [ "index_name-*" ], "settings": { "number_of_shards": 1, "number_of_replicas": 0 "index.lifecycle.name": "new_policy", "index.lifecycle.rollover_alias": "new_index" }, "mappings": {<....> } } ``` 3 rd step- ```auto PUT index-name-000001 { "aliases": { "new_index": { "is_write_index": true } } } ``` ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 9, 2022, 2:01pm UTC](https://discuss.elastic.co/t/automate-deletion-of-docs-from-an-index-that-are-older-than-10-days/293826/2 "2022-01-09T14:01:24Z")
ILM requires time-based indices so that complete indices can be deleted once indices exceed the retention period. This is much more efficient than using delete-by-query to periodically trim indices, which is why it is the recommended approach. If you wish to continue relying on DBQ ILM will not be able to help you and you need to continue running periodic jobs yourself. ---
**Author:** ![Divyank\_Mahalle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/divyank_mahalle/32/91240_2.png) [@Divyank\_Mahalle](https://discuss.elastic.co/u/Divyank_Mahalle)\ **Post date:** [January 21, 2022, 11:16am UTC](https://discuss.elastic.co/t/automate-deletion-of-docs-from-an-index-that-are-older-than-10-days/293826/3 "2022-01-21T11:16:31Z")
Thanks @Christian_Dahlqvist , My index was not time series index, post reindexing it I convert it into time series index and rollover is happening as per ilm policy and able to achieve the goal. ```auto POST /_reindex { "source": { "index": "weather" }, "dest": { "index": "weather-000001" } } DELETE weather Appying Alias on timeseries index, Queries can run on alias which will be the weather. POST /_aliases { "actions": [ { "add": { "index": "weather-000001", "alias": "weather", "is_write_index": true } } ] } ``` Logstash output- ```auto output { stdout { codec => rubydebug } elasticsearch { ilm_rollover_alias => "weather" ilm_pattern => "000001" ilm_policy => "weather" hosts => ["localhost:9200"] ``` **One Query-** If we query (using an index pattern) for the last 3 days of data in an index that has rollover set to daily, does the query run only on the latest 3 indices or does it run on all? I guess we need to specify the indices names in the query to limit to search to only the last 3 indices. Doesn't it happen automatically? Or does it happen automatically in the data stream but not in the index alias/ pattern? If we have 10 days of data with daily rollover. Sample query- ```auto GET weather-*/_search { "query": { "range": { "@timestamp": { "gte": "now-3d" } } } } ``` Ref link- [ ![](https://us1.discourse-cdn.com/elastic/optimized/3X/5/d/5ded4d26a9eb0b035f2650d43025d92144c19f9c_2_32x32.ico)Index lifecycle error - illegal\_argument\_exception: index.lifecycle.rollover\_alias](https://discuss.elastic.co/t/index-lifecycle-error-illegal-argument-exception-index-lifecycle-rollover-alias/207900/8) [![](https://us1.discourse-cdn.com/elastic/original/3X/9/2/92fb7ff93b9877fd59341d159284e1a247bb277b.png)Manage existing indices | Elasticsearch Guide [7.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.15/ilm-with-existing-indices.html#ilm-existing-indices-apply) [![](https://us1.discourse-cdn.com/elastic/original/3X/d/f/df9825935fc1b209109e6aad25db8f47d93c762e.png)Data rollover in Elasticsearch](https://medium.com/nerd-for-tech/data-rollover-in-elasticsearch-b809bb9f150a) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 21, 2022, 11:20am UTC](https://discuss.elastic.co/t/automate-deletion-of-docs-from-an-index-that-are-older-than-10-days/293826/4 "2022-01-21T11:20:33Z")
You can generally query all indices matching a specific pattern as Elasticsearch nowadays will rewrite the query so that indices not holding any data matching the time interval are not queried. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 18, 2022, 11:21am UTC](https://discuss.elastic.co/t/automate-deletion-of-docs-from-an-index-that-are-older-than-10-days/293826/5 "2022-02-18T11:21:09Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.