# Daily index maintenance **URL:** \ **Category:** Elasticsearch\ **Created:** [May 7, 2018, 8:40pm UTC](https://discuss.elastic.co/t/daily-index-maintenance/130902 "2018-05-07T20:40:45Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ankitachow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitachow/32/46246_2.png) [@ankitachow](https://discuss.elastic.co/u/ankitachow)\ **Post date:** [May 7, 2018, 8:40pm UTC](https://discuss.elastic.co/t/daily-index-maintenance/130902/1 "2018-05-07T20:40:46Z")
Hi, In my current application, I'm rolling over data with a doc limit. I'm planning to implement the logic of creating daily indices in my platform. I would like to know how would daily index affect below parameters: 1. Processing old data. For eg: in my application, I expect old data with a latency of XX hrs. So, once the new index is created with current date, I can still expect some yesterday's data. But since my current day's index is active, how can I maintain yesterday's data in m current index. Retention will also be a problem in this scenario. 2. Creating daily index, how the response time for search be affected? How to ensure the request goes only to few shards of the day for which the request was issued. I'm planning to follow the below link: > **[And the big one said "Rollover" — Managing Elasticsearch time-based indices...](https://www.elastic.co/blog/managing-time-based-indices-efficiently)** > > Introducing the new Rollover Pattern, and the APIs which support it, which is a simpler, more efficient way of managing time-based indices in Elasticsearch. Thanks Ankita ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 8, 2018, 6:40am UTC](https://discuss.elastic.co/t/daily-index-maintenance/130902/2 "2018-05-08T06:40:16Z")
1. you'd need to use time based indices and then have your processing layer send the delayed data to the older index. 2. Depends on how you are querying the indices . ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 8, 2018, 6:59am UTC](https://discuss.elastic.co/t/daily-index-maintenance/130902/3 "2018-05-08T06:59:22Z")
> [@ankitachow](#): > > Creating daily index, how the response time for search be affected? How to ensure the request goes only to few shards of the day for which the request was issued. Kibana used to limit the indices being queried, first by using date match based on the timestamp in the index name and later based on field stats. Improvements in Elasticsearch has meant that this is no longer required, and Kibana now sends the query to all shards matching the index pattern, so if you are on version 6.x you may not need to worry about this. ---
**Author:** ![ankitachow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ankitachow/32/46246_2.png) [@ankitachow](https://discuss.elastic.co/u/ankitachow)\ **Post date:** [May 10, 2018, 2:44pm UTC](https://discuss.elastic.co/t/daily-index-maintenance/130902/4 "2018-05-10T14:44:55Z")
@warkolm: How can the processing layer process be made t process old data and send to older index? From what I understand, the daily indexes that are created are not on the contents of the data but the time when the data has been processed. for eg: if data is processed on 5/10 it will do to 5/10's index even when it has 5/9's data. Also, when I'm using alias, with a rollover is happening on the on the daily index, the wite alias points to the current and so writing data to the old index is also a challenge. How to overcome that? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 11, 2018, 5:16am UTC](https://discuss.elastic.co/t/daily-index-maintenance/130902/5 "2018-05-11T05:16:55Z")
If you are using Logstash then it'll automatically pick the right day's index as long as you have a date filter taking the event date from the event. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 11, 2018, 6:30am UTC](https://discuss.elastic.co/t/daily-index-maintenance/130902/6 "2018-05-11T06:30:10Z")
If you are using rollover, events that are processed late and indexed into the write alias will indeed end up in the current index. You could based on index statistics determine which index/indices the data should go to and index directly to the indices rather than through the alias, but there is no automatic way to do so. If you have data coming in very late, it may be better for you to stick with indices matching fixed time periods based on the index name. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 8, 2018, 6:32am UTC](https://discuss.elastic.co/t/daily-index-maintenance/130902/7 "2018-06-08T06:32:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.