In my current application, I'm rolling over data with a doc limit. I'm planning to implement the logic of creating daily indices in my platform. I would like to know how would daily index affect below parameters:
Processing old data. For eg: in my application, I expect old data with a latency of XX hrs. So, once the new index is created with current date, I can still expect some yesterday's data. But since my current day's index is active, how can I maintain yesterday's data in m current index. Retention will also be a problem in this scenario.
Creating daily index, how the response time for search be affected? How to ensure the request goes only to few shards of the day for which the request was issued.
I'm planning to follow the below link: