# Over-allocation of shards **URL:** \ **Category:** Elasticsearch\ **Created:** [January 19, 2012, 10:40pm UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448 "2012-01-19T22:40:33Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [January 19, 2012, 10:40pm UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/1 "2012-01-19T22:40:33Z")
Hi, Shay mentioned/suggested over-allocation of shards on a few occasions and I wanted to make sure I understand why. Is this primarily to accommodate future cluster growth, so that new nodes can be added and existing shards re-distributed / re-balanced over all old+new nodes? Or is there some other or additional reason for this? ## Thanks, Otis Sematext is Hiring World-Wide -- [http://sematext.com/about/jobs.html](http://sematext.com/about/jobs.html) ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 19, 2012, 11:02pm UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/2 "2012-01-19T23:02:22Z")
Yes, overallocation of shards is to accommodate cluster growth in certain "data flows". For example, by default, when you create an index with elasticsearch, it has 5 shards by default, even if you use just one node. This allows the index to grow up to 5 nodes (assuming no replicas and no other indices, so you have 1 shard per node) in terms of size (read perf can be increased by increasing replicas (and nodes)). Going back to "data flow". This is the one of the most important concept to understand about your system when you use elasticsearch. Time based data (tweets, logs) can be indexed using rolling time base indices, and then, the number of shards per index are simply there so accomodate the expected size for that time frame the index is responsible for. Aliases can be used then to point to "latest" index to index to, or time base aliases can be created that point to several indices to create "bigger" time frames without needing to specify all the indices that compose that time frame. A "user" based data flow, in theory, is perfect for an index per user case. If you have enough nodes (each shard is a Lucene index, which has a cost) in the cluster to do that, thats great, and several very large scale ES users actually do that. But, if you have a small cluster (or constraint budget / HW wise), you could say, ok, I will do all on a single index, but then, that index will need quite a few shards to support potential growth. This can be problematic without doing anything else, because if you create an index with 30 shards on a single node cluster, each search request (for that user data) will need to span all 30 shards. This is where routing comes to play. You can have all the data for a user allocated to a specific shard using routing (with clever use of routing, you can actually have user data span several shards, for example, using time base routing included in the user routing value). This means that when you search for that user data, and provide the routing value, it will only go to a single shard. You can create an index with 100 shards, and you won't incur the overhead of searching across all shards for specific user data. Obviously, when you do that, you also need to filter each query to only have that specific user data. Remember also, that you can still search across several users data by simply using several routing values. For this usecase, aliases really shine. You can have a single index, lets call it users, with 50 shards. And define an alias for each user. Lets say the first user is user1, you define an alias called user1. That alias can have a routing value of "user1", which means when you index and search against that alais (as if it was an "index', i.e. /user1/\_search), the routing value will be automatically applied. But, you still need to filter to only see that user data. For that, an alias can also be associated with a filter (i.e. a term filter on the user name with the user value). Then, every time you search against /user1/\_search, the filter will be automatically applied. One nice aspect of it is that you can also search across several aliases. For example, /user1,user2/\_search (as you can search over multiple indices), and it will "do the right thing". It will do a search with two routing values, and an "OR'ed" filter. -shay.banon On Fri, Jan 20, 2012 at 12:40 AM, Otis Gospodnetic \< [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > Hi, > > Shay mentioned/suggested over-allocation of shards on a few occasions > and I wanted to make sure I understand why. > > Is this primarily to accommodate future cluster growth, so that new > nodes can be added and existing shards re-distributed / re-balanced > over all old+new nodes? > > Or is there some other or additional reason for this? > > ## Thanks, Otis > > Sematext is Hiring World-Wide -- [Jobs - Sematext](http://sematext.com/about/jobs.html) ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [January 20, 2012, 12:51am UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/3 "2012-01-20T00:51:31Z")
On Jan 19, 3:02 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Yes, overallocation of shards is to accommodate cluster growth in certain > "data flows". For example, by default, when you create an index with > elasticsearch, it has 5 shards by default, even if you use just one node. > This allows the index to grow up to 5 nodes (assuming no replicas and no > other indices, so you have 1 shard per node) in terms of size (read perf > can be increased by increasing replicas (and nodes)). I've wondered about the default setting of 5 shards as well. If I know how much data I'm indexing (e.g. indexing an archive of tweets), I'll want to change that setting. If I don't (e.g. indexing tweets in "realtime"), I want to start with 1 shard (good enough for most users), and re-shard to more than that only when the number of documents passes a threshold. This requires rebuilding the index; would be nice if elasticsearch had a simple command to do that. ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [January 20, 2012, 3:03am UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/4 "2012-01-20T03:03:04Z")
Thank you Shay for a thorough answer - we've been making use of exactly these index alias/sharding/routing functionality in a number of projects where we used Elasticsearch. As a matter of fact, this is from 4 hours ago: [http://twitter.com/#!/otisg/status/160128277497913345](http://twitter.com/#!/otisg/status/160128277497913345) And thanks for confirming the over-allocation bit! ## Otis Sematext is Hiring World-Wide -- [Jobs](http://sematext.com/about/jobs.html) On Jan 19, 6:02 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Yes, overallocation of shards is to accommodate cluster growth in certain > "data flows". For example, by default, when you create an index with > elasticsearch, it has 5 shards by default, even if you use just one node. > This allows the index to grow up to 5 nodes (assuming no replicas and no > other indices, so you have 1 shard per node) in terms of size (read perf > can be increased by increasing replicas (and nodes)). > > Going back to "data flow". This is the one of the most important concept to > understand about your system when you use elasticsearch. Time based data > (tweets, logs) can be indexed using rolling time base indices, and then, > the number of shards per index are simply there so accomodate the expected > size for that time frame the index is responsible for. Aliases can be used > then to point to "latest" index to index to, or time base aliases can be > created that point to several indices to create "bigger" time frames > without needing to specify all the indices that compose that time frame. > > A "user" based data flow, in theory, is perfect for an index per user case. > If you have enough nodes (each shard is a Lucene index, which has a cost) > in the cluster to do that, thats great, and several very large scale ES > users actually do that. But, if you have a small cluster (or constraint > budget / HW wise), you could say, ok, I will do all on a single index, but > then, that index will need quite a few shards to support potential growth. > This can be problematic without doing anything else, because if you create > an index with 30 shards on a single node cluster, each search request (for > that user data) will need to span all 30 shards. > > This is where routing comes to play. You can have all the data for a user > allocated to a specific shard using routing (with clever use of routing, > you can actually have user data span several shards, for example, using > time base routing included in the user routing value). > > This means that when you search for that user data, and provide the routing > value, it will only go to a single shard. You can create an index with 100 > shards, and you won't incur the overhead of searching across all shards for > specific user data. Obviously, when you do that, you also need to filter > each query to only have that specific user data. Remember also, that you > can still search across several users data by simply using several routing > values. > > For this usecase, aliases really shine. You can have a single index, lets > call it users, with 50 shards. And define an alias for each user. Lets say > the first user is user1, you define an alias called user1. That alias can > have a routing value of "user1", which means when you index and search > against that alais (as if it was an "index', i.e. /user1/\_search), the > routing value will be automatically applied. But, you still need to filter > to only see that user data. For that, an alias can also be associated with > a filter (i.e. a term filter on the user name with the user value). Then, > every time you search against /user1/\_search, the filter will be > automatically applied. > > One nice aspect of it is that you can also search across several aliases. > For example, /user1,user2/\_search (as you can search over multiple > indices), and it will "do the right thing". It will do a search with two > routing values, and an "OR'ed" filter. > > -shay.banon > > On Fri, Jan 20, 2012 at 12:40 AM, Otis Gospodnetic \< > > [otis.gospodne...@gmail.com](mailto:otis.gospodne...@gmail.com)\> wrote: > > > Hi, > > > Shay mentioned/suggested over-allocation of shards on a few occasions > > and I wanted to make sure I understand why. > > > Is this primarily to accommodate future cluster growth, so that new > > nodes can be added and existing shards re-distributed / re-balanced > > over all old+new nodes? > > > Or is there some other or additional reason for this? > > > ## Thanks, Otis > > > > Sematext is Hiring World-Wide --[Jobs](http://sematext.com/about/jobs.html) ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [January 20, 2012, 11:00am UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/5 "2012-01-20T11:00:04Z")
Hi Eric > I've wondered about the default setting of 5 shards as well. If I know > how much data I'm indexing (e.g. indexing an archive of tweets), I'll > want to change that setting. You can specify that when creating the index > If I don't (e.g. indexing tweets in > "realtime"), I want to start with 1 shard (good enough for most > users), and re-shard to more than that only when the number of > documents passes a threshold. This requires rebuilding the index; > would be nice if elasticsearch had a simple command to do that. you can use several indices and create a new index if documents passes a threshold (+ use the index alias feature) ... No need to recreate shards ... Peter. ---
**Author:** ![Eric\_Jain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eric_jain/32/834_2.png) [@Eric\_Jain](https://discuss.elastic.co/u/Eric_Jain)\ **Post date:** [January 20, 2012, 7:16pm UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/6 "2012-01-20T19:16:45Z")
On Fri, Jan 20, 2012 at 03:00, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com) wrote: > > I've wondered about the default setting of 5 shards as well. If I know > > how much data I'm indexing (e.g. indexing an archive of tweets), I'll > > want to change that setting. > > You can specify that when creating the index That's what I do; just pointing out that I never had a case where I wanted to start an index with precisely 5 shards. I'm assuming that queries on an index with 5 shards running on a single machine are going to be slower and use more resources than queries on an index with a single shard, right? > you can use several indices and create a new index if documents > passes a threshold (+ use the index alias feature) ... > No need to recreate shards ... Hadn't considered that approach! I imagine that internally, shards are similar to aliased indexes? So if there was a simple way to replace "partition by hash" with "partition by row count" (and have shards created lazily)... ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [January 21, 2012, 8:51pm UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/7 "2012-01-21T20:51:34Z")
> on a single machine are going to be slower yes, of course. > Hadn't considered that approach! I imagine that internally, shards are > similar to aliased indexes? So if there was a simple way to replace > "partition by hash" with "partition by row count" (and have shards > created lazily)... there is a similar feature (not really what you are after) .. via routing. but creating 'shards' lazily can only be done via creating 'indices' which are under your control. Peter. ---
**Author:** ![chenry12](https://avatars.discourse-cdn.com/v4/letter/c/f0a364/32.png) [@chenry12](https://discuss.elastic.co/u/chenry12)\ **Post date:** [March 24, 2013, 8:41pm UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/8 "2013-03-24T20:41:41Z")
Hi, When you use this method : 1 index + routing. It is not possible to apply an analyser by alias as when we work with an index, isnt it ? I have an other question: I have listened the Shay Banon's presentation [http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html](http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html) and I have a question: I dont understand exactly the difference to build 1 shard per index and this method to build 1 index + routing. I understood that it's better for the system ressource but not exactly why ? Thanks in advance, Christophe. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![chenry12](https://avatars.discourse-cdn.com/v4/letter/c/f0a364/32.png) [@chenry12](https://discuss.elastic.co/u/chenry12)\ **Post date:** [March 25, 2013, 7:14am UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/9 "2013-03-25T07:14:04Z")
Ok, i understood my second question. More of one user can use a same shard. And are you ok with my first question ? Thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:44am UTC](https://discuss.elastic.co/t/over-allocation-of-shards/6448/10 "2017-07-06T02:44:50Z")