# Rolling indices daily best practices? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 24, 2015, 3:59pm UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466 "2015-11-24T15:59:06Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![JeremyinNC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremyinnc/32/44868_2.png) [@JeremyinNC](https://discuss.elastic.co/u/JeremyinNC)\ **Post date:** [November 24, 2015, 3:59pm UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466/1 "2015-11-24T15:59:06Z")
We are using ES as a data store for events from devices. By the end of the year I expect to have a few 100 million events being written every day. My plan is to create two aliases which will get used by clients: - An alias ("events-current") that points to the current day's index - Another ("events-all") that contains all of the event indices. To do this I am planning to create a script that will: - Export the mappings from the index behind events-current - Create a daily index "events-YYYY.MM.DD" - Apply the mappings from the previous day's index to the new index - Moving removing the previous day's index from the "events-current" alias - Adding the new index to both the "events-current and "events-all" aliases I can do this with shell scripts but there has to be a better way. I'm pretty sure I'm doing the same thing that logstash does by default but wanted to know if I'm missing anything or anyone had a suggestion of a better way to set this up. ---
**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\ **Post date:** [November 24, 2015, 5:15pm UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466/2 "2015-11-24T17:15:05Z")
You can read Curator plugin: [https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) "Apply the mappings from the previous day's index to the new index": use templates [https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html) But you will still have to do the "CRON" task yourself. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 25, 2015, 1:00am UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466/3 "2015-11-25T01:00:53Z")
Why not just use a template instead? ---
**Author:** ![JeremyinNC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremyinnc/32/44868_2.png) [@JeremyinNC](https://discuss.elastic.co/u/JeremyinNC)\ **Post date:** [December 7, 2015, 6:42pm UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466/4 "2015-12-07T18:42:37Z")
Does the template control when to create new indices? I thought it just defined the settings for a specific index. ---
**Author:** ![JeremyinNC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremyinnc/32/44868_2.png) [@JeremyinNC](https://discuss.elastic.co/u/JeremyinNC)\ **Post date:** [December 7, 2015, 6:45pm UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466/5 "2015-12-07T18:45:14Z")
Sorry for the late response, was out of town last week. The mappings may change from day to day, when a device registers it is able to define a semi-arbitrary set of events that it supports, if I used templates I'd have to update the template every time I added a new type of event. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 7, 2015, 7:00pm UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466/6 "2015-12-07T19:00:22Z")
Templates do support pattern matching to some degree but I see your point. > [@JeremyinNC](#): > > semi-arbitrary I should mention that you should pay attention to the total number of fields in your indexes. Each one has overhead and letting things just create whatever they want might destabilize things. Its fine if devices don't create more than they need but you'll be vulnerable to mistakes like uuids as field names. ---
**Author:** ![JeremyinNC](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jeremyinnc/32/44868_2.png) [@JeremyinNC](https://discuss.elastic.co/u/JeremyinNC)\ **Post date:** [December 7, 2015, 7:06pm UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466/7 "2015-12-07T19:06:45Z")
Thanks for the feedback! I do think I could do something like 1. Export the existing index's mapping as a template 2. Create ther new index & make the alias changes and it would at least save one step. We are already using templates to explicitly set some field types we're we've had some confusion of the type of data (boolean/string etc). There should not be _that_ many event types, maybe 100 or so all together. Longer term we're going to split the indexes by device type as well. My main concern is i don't want to be manually updating the templates every time we add a device or new functionality to an existing device. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 7, 2015, 7:10pm UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466/8 "2015-12-07T19:10:28Z")
> [@JeremyinNC](#): > > My main concern is i don't want to be manually updating the templates every time we add a device or new functionality to an existing device. Yeah, manual would be a pain. When I maintained some production indexes I used `"dynamic": false` and added the new properties with a script. I just didn't want anything sneaking in properties I didn't know about. But I added properties much less frequently than it sounds like you will. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:32pm UTC](https://discuss.elastic.co/t/rolling-indices-daily-best-practices/35466/9 "2017-07-05T23:32:59Z")