# Tradeoffs for using week/month (time) based indices **URL:** \ **Category:** Elasticsearch\ **Created:** [January 23, 2013, 7:19am UTC](https://discuss.elastic.co/t/tradeoffs-for-using-week-month-time-based-indices/10467 "2013-01-23T07:19:58Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![T\_Vinod\_Gupta](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@T\_Vinod\_Gupta](https://discuss.elastic.co/u/T_Vinod_Gupta)\ **Post date:** [January 23, 2013, 7:19am UTC](https://discuss.elastic.co/t/tradeoffs-for-using-week-month-time-based-indices/10467/1 "2013-01-23T07:19:58Z")
hi, in couple of talks on ES performance (including the one by shay), it was illustrated that for constantly growing data streams (like facebook/twitter or log streams) it is better to create indices for each week/month. and then an aliased index can be created to combine a bunch of them into a logical index. this will help in keeping the searchable space under control and avoid heavy deletions. but then at the same time, it also means that any search would be hitting multiple shards and that has a perf penalty. one can use routing parameter to direct all traffic for a user to 1 shard. that way searches are fast. but using multiple indices kind of offsets it.. im little confused/unclear now. does anyone have any insights? thanks -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 23, 2013, 10:01am UTC](https://discuss.elastic.co/t/tradeoffs-for-using-week-month-time-based-indices/10467/2 "2013-01-23T10:01:01Z")
Hello, In the example of facebook/twitter/logs, newer data is generally "hotter". And you can put some default time limits in your application so that most searches will hit just the newest index. And because that index is small, the majority of your searches will turn out to be faster. And because the "current" index always changes, you don't need to overshard. For example, if you have 8 nodes, you can have 8 shards (or 4 shards and 1 replica, etc), and use Shard Allocation[0] to make sure your index is well distributed in your cluster. Then if you add a 9th node, you can change the settings/template to make the next "current" index have 9 shards. Of course, users should be able to widen their searches and hit more/all indices - and in that case there's indeed a performance penalty like you said. But in regular usecases for time-based indices, you can assume that old ones will be more or less unchanged, so you can optimize "old" indices and searches on them should be pretty fast. Then there's indexing performance. Because the "current" index is smaller, there's going to be less merging while indexing - and that will put less strain on I/O. [0] > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Wed, Jan 23, 2013 at 9:19 AM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > hi, > in couple of talks on ES performance (including the one by shay), it was > illustrated that for constantly growing data streams (like facebook/twitter > or log streams) it is better to create indices for each week/month. and > then an aliased index can be created to combine a bunch of them into a > logical index. this will help in keeping the searchable space under control > and avoid heavy deletions. > > but then at the same time, it also means that any search would be hitting > multiple shards and that has a perf penalty. one can use routing parameter > to direct all traffic for a user to 1 shard. that way searches are fast. > but using multiple indices kind of offsets it.. > > im little confused/unclear now. > does anyone have any insights? > > thanks > > -- -- ---
**Author:** ![T\_Vinod\_Gupta](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@T\_Vinod\_Gupta](https://discuss.elastic.co/u/T_Vinod_Gupta)\ **Post date:** [January 23, 2013, 8:28pm UTC](https://discuss.elastic.co/t/tradeoffs-for-using-week-month-time-based-indices/10467/3 "2013-01-23T20:28:32Z")
Thanks Radu for the elaborate response. I am wondering if someone has done experiments on what is a good partition size for a shard on a m1.xlarge host on ec2? how many docs/data size is good for 1 shard? for instance, i currently have about 20M docs in a 5 shard index on an m1.xlarge. this is with no routing intelligence.. so the searches go to all shards.. and my searches take over 10s of seconds many times.. but my last month data is only 2.3M docs. so if i create a month index with just 1 shard, will that suffice? or i create 2 shards for the month data with routing in it. my typical search is over 3 months. so in the month based scheme, i would be going to 3 shards instead of earlier 5 and on smaller shards. any insights will be helpful. thanks On Wed, Jan 23, 2013 at 2:01 AM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com)wrote: > Hello, > > In the example of facebook/twitter/logs, newer data is generally "hotter". > And you can put some default time limits in your application so that most > searches will hit just the newest index. And because that index is small, > the majority of your searches will turn out to be faster. > > And because the "current" index always changes, you don't need to > overshard. For example, if you have 8 nodes, you can have 8 shards (or 4 > shards and 1 replica, etc), and use Shard Allocation[0] to make sure your > index is well distributed in your cluster. Then if you add a 9th node, you > can change the settings/template to make the next "current" index have 9 > shards. > > Of course, users should be able to widen their searches and hit more/all > indices - and in that case there's indeed a performance penalty like you > said. But in regular usecases for time-based indices, you can assume that > old ones will be more or less unchanged, so you can optimize "old" indices > and searches on them should be pretty fast. > > Then there's indexing performance. Because the "current" index is smaller, > there's going to be less merging while indexing - and that will put less > strain on I/O. > > [0] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/allocation.html) > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Wed, Jan 23, 2013 at 9:19 AM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > > > hi, > > in couple of talks on ES performance (including the one by shay), it was > > illustrated that for constantly growing data streams (like facebook/twitter > > or log streams) it is better to create indices for each week/month. and > > then an aliased index can be created to combine a bunch of them into a > > logical index. this will help in keeping the searchable space under control > > and avoid heavy deletions. > > > > but then at the same time, it also means that any search would be hitting > > multiple shards and that has a perf penalty. one can use routing parameter > > to direct all traffic for a user to 1 shard. that way searches are fast. > > but using multiple indices kind of offsets it.. > > > > im little confused/unclear now. > > does anyone have any insights? > > > > thanks > > > > -- > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:54am UTC](https://discuss.elastic.co/t/tradeoffs-for-using-week-month-time-based-indices/10467/4 "2017-07-06T02:54:55Z")