# Pro and cons between using daily and monthly indexes? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 20, 2021, 10:19am UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613 "2021-09-20T10:19:15Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Post date:** [September 20, 2021, 10:19am UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/1 "2021-09-20T10:19:16Z")
Hello Following the advice given here (thank you), in a few months, Ill be able to reduce the total shards in my Elastic Stack single node from aprox 5000 to 160. This will generally give it better performance and not overload the single node. I do have worries still though 🙂 What are some pros and cons to changing from daily to monthly indexes? Thank you ---
**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Post date:** [September 20, 2021, 10:20am UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/2 "2021-09-20T10:20:32Z")
Before anyone asks why not merge them now..... The client asks for a retention period of 180 days; If I merge them, the new index gets a creation date of today, not months ago, breaking the ILP of deleting after x days of index creation. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 11:24am UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/3 "2021-09-20T11:24:36Z")
You could reindex the old data in indices named `data-2021-06`, `data-2021-07`, `data-2021-08`, `data-2021-09`... That would work well, no? I see the problem with ILM though. I think you can use `index.lifecycle.origination_date` ([Index lifecycle management settings in Elasticsearch | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/ilm-settings.html#index-lifecycle-origination-date)). Read also [Manage existing indices | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/ilm-with-existing-indices.html#ilm-existing-indices-apply) ---
**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Post date:** [September 20, 2021, 12:20pm UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/4 "2021-09-20T12:20:53Z")
> You could reindex the old data in indices named `data-2021-06` , `data-2021-07` , `data-2021-08` , `data-2021-09` ... > That would work well, no? > I see the problem with ILM though. I think you can use `index.lifecycle.origination_date` ([Index lifecycle management settings in Elasticsearch | Elasticsearch Guide [7.14] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.14/ilm-settings.html#index-lifecycle-origination-date)). The thing is this needs to be supported in the Merge API, right? When I do the merge, I set index.lifecycle.origination\_date to the epoch date, it will create the NEW index with that date/time correct? ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 2:04pm UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/5 "2021-09-20T14:04:15Z")
> [@riahc3](#): > > The thing is this needs to be supported in the Merge API, right? What is the "Merge API"? ---
**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Post date:** [September 20, 2021, 2:06pm UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/6 "2021-09-20T14:06:32Z")
> **[Force merge API | Elasticsearch Guide \[master\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/master/indices-forcemerge.html)** ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 2:19pm UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/7 "2021-09-20T14:19:50Z")
It's not related I believe. force merge is just merging segments within a shard of a given index. It does not merge "indices" if this is what you are looking for. ---
**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Post date:** [September 20, 2021, 2:35pm UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/8 "2021-09-20T14:35:28Z")
Ah, my mistake. I ment the reindex API At a quick glance, it doesnt seem to support the index.lifecycle.origination\_date paramter. (BTW, we are getting offtopic here) ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 3:10pm UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/9 "2021-09-20T15:10:23Z")
So the reindex API reads data from one or many indices using the Scroll API and then load them into a destination index using the Bulk API. The destination index is like any normal index. I mean that you can create it and set `index.lifecycle.origination_date` setting IMO. Then call the reindex API. ---
**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Post date:** [September 20, 2021, 3:17pm UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/10 "2021-09-20T15:17:27Z")
OK, so if I understand correctly..... I should copy a ILM policy i use BUT set a index.lifecycle.origination\_date. Create a blank empty index using that new ILM policy Copy from my old index (with the original ILM policy) to the new index (with the copied ILM policy) and you are saying that the index creation date on that new one should be the one I established? It sounds intresting......not exactly sure if it works like that. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 20, 2021, 4:22pm UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/11 "2021-09-20T16:22:23Z")
It would be great if you can try to reproduce this scenario in a test env and make sure it works as you'd expect. And update this thread so the community will benefit from your experience 🙂 ---
**Author:** ![riahc3](https://avatars.discourse-cdn.com/v4/letter/r/d6d6ee/32.png) [@riahc3](https://discuss.elastic.co/u/riahc3)\ **Post date:** [September 21, 2021, 8:47am UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/12 "2021-09-21T08:47:32Z")
As you problably well know, production IS the test scenario 😉 but sadly, it is for a external client so I cannot do it. It does look good on paper (documentation) and should work. Plus building a Powershell script for it shouldnt be THAT difficult. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 19, 2021, 8:48am UTC](https://discuss.elastic.co/t/pro-and-cons-between-using-daily-and-monthly-indexes/284613/13 "2021-10-19T08:48:27Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.