# Delete 7 days old data everyday **URL:** \ **Category:** Elasticsearch\ **Created:** [November 13, 2018, 8:54am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418 "2018-11-13T08:54:11Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 13, 2018, 8:54am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/1 "2018-11-13T08:54:12Z")
Hi I want to delete 7 days old data in my index everyday. I have not used time based index. In that scenario, will curator help? If so, I prefer to write java scheduler,can i write a java code using curator? Or Is it only python? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 13, 2018, 9:29am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/2 "2018-11-13T09:29:00Z")
That will be very inefficient but you can call DELETE BY QUERY API from a script that you can put in your crontab or from any Java app you want. Using daily indices and dropping them every 7 days will be much much more efficient. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 13, 2018, 9:43am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/3 "2018-11-13T09:43:57Z")
In that case, this is my scenario, I will create two indices during installation. All data will get loaded there. First index need to purge 7 days old data. Second index purging time frame depends on user input. Every type in second index will have a column which will tell days to retain. How to use time based indices for second index? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 13, 2018, 9:59am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/4 "2018-11-13T09:59:12Z")
> Every type in second index will have a column which will tell days to retain. Instead of sending the data to `index-foo`, send it to `index-foo-date-to-retain` like `index-foo-2018-11-20`. On day `20/11/2018`, drop all indices where name is `index-foo-2018-11-20` for example. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 13, 2018, 10:06am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/5 "2018-11-13T10:06:58Z")
In case of reindexing/migration, how to migrate all data? Should I migrate data from all indices? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 13, 2018, 10:32am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/6 "2018-11-13T10:32:26Z")
You can use the reindex API probably. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 13, 2018, 11:23am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/7 "2018-11-13T11:23:36Z")
Can I use alias in case of time based index? Basically I need a fixed name for my index to refer in java /kibana ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 13, 2018, 11:42am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/8 "2018-11-13T11:42:24Z")
Yes you can. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 13, 2018, 2:15pm UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/9 "2018-11-13T14:15:35Z")
can you please refer me a documentation to create time based indices using java API(elasticsearch 5.6.3)? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [November 13, 2018, 4:09pm UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/10 "2018-11-13T16:09:28Z")
Not really. That's just a question of index naming... So using [https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-create-index.html](https://www.elastic.co/guide/en/elasticsearch/client/java-rest/current/java-rest-high-create-index.html) ```auto CreateIndexRequest request = new CreateIndexRequest("twitter-2018-11-13"); ``` Just make the index name based on the current time... And use index templates (which is not something you must do using the Java API) ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [November 15, 2018, 10:44am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/11 "2018-11-15T10:44:32Z")
@RAM_NATHAN, as per my knowledge you need to create one indices per day with date timestamp. Then you can use curator to delete indices in very effective way. I am already using curator to delete 20 old days indices automatically from elasticsearch. Thanks. ---
**Author:** ![gavenkoa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gavenkoa/32/23899_2.png) [@gavenkoa](https://discuss.elastic.co/u/gavenkoa)\ **Post date:** [November 18, 2018, 11:18am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/12 "2018-11-18T11:18:46Z")
DELETE BY QUERY API just marks records as deleted. It doesn't reduce index size or data size. You can reindex your data though but it is unusual usage pattern of ES. You made problem from nothing )) ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 19, 2018, 5:17am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/13 "2018-11-19T05:17:17Z")
sorry I'm not clear. Our product is in production with Kibana 1.x. Now we are migrating to 5.6.x. Creating time based indices would need impact analysis and other checks. So we thought as of now we will stick to DELETE BY QUERY API. But if it wont get deleted, I dont know why this API in place. ---
**Author:** ![gavenkoa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gavenkoa/32/23899_2.png) [@gavenkoa](https://discuss.elastic.co/u/gavenkoa)\ **Post date:** [November 19, 2018, 9:16am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/14 "2018-11-19T09:16:58Z")
It is here (delete API) because people need to delete some documents, to no longer see them in search responses. If you search across product line - you don't have another option - you need to delete + reindex. If you work with time series it is cumbersome and you need to go with time based indexes and dropping old unused indexes. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 19, 2018, 9:21am UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/15 "2018-11-19T09:21:21Z")
thanks for the reply. Whats the solution then?? I need to purge data once in a while ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 26, 2018, 12:19pm UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/16 "2018-11-26T12:19:03Z")
Can you please reply? Using time based indices is the only option to delete the data?? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 26, 2018, 12:29pm UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/17 "2018-11-26T12:29:13Z")
When you are using delete-by-query to manage retention, you require a lot more processing that if you use time-base indices. As you typically also tend to delete the oldest data, which tends to be located in the largest and oldest segments. The data will actually only be removed from disk once these segments are merged and this can time as a lot of data in these segments need to get deleted before they are subject to merging. You can get around this by explicitly issuing a force merge command after the delete, but this is also a quite expensive operation. The conclusion is that by not using time-based indices you require a lot of expensive extra processing that will prevent you from getting the most from your cluster. ---
**Author:** ![RAM\_NATHAN](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ram_nathan/32/50393_2.png) [@RAM\_NATHAN](https://discuss.elastic.co/u/RAM_NATHAN)\ **Post date:** [November 26, 2018, 12:35pm UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/18 "2018-11-26T12:35:01Z")
Thanks. I misunderstood earlier comments as, DELETE\_BY\_QUERY not at all deletes the data. So I'm taking this as DELETE\_BY\_QUERY deletes the data,but it took lot of processing, at the end can cause performance issues ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [November 26, 2018, 12:36pm UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/19 "2018-11-26T12:36:28Z")
It deletes the data, but that does not necessarily mean that disk space will be freed up immediately. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 24, 2018, 12:36pm UTC](https://discuss.elastic.co/t/delete-7-days-old-data-everyday/156418/20 "2018-12-24T12:36:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.