# When is one shard enough **URL:** \ **Category:** Elasticsearch\ **Created:** [December 12, 2013, 4:13pm UTC](https://discuss.elastic.co/t/when-is-one-shard-enough/14840 "2013-12-12T16:13:53Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 12, 2013, 4:13pm UTC](https://discuss.elastic.co/t/when-is-one-shard-enough/14840/1 "2013-12-12T16:13:53Z")
I maintain search indexes for a bunch of wikis (hundreds), many of which are small (hundreds of documents), some of which are pretty big (a few million documents). I've noticed that having five shards per index for the small wikis seems like overkill. It feels like the cluster is doing more work maintaining index metadata for some wikis then it is for lucene stuff. OTOH, my master node still has very low CPU usage. Does anyone run with one or two shards for small indexes? Caveats: 1. Technically I split each wiki into two indexes to make the suggestion results better. The split ends up putting about 3/4 of the documents in one index and 1/4 in the other. In any case, that means even more shards with even less data in them for the small wikis. 2. Update and search rate is mostly proportional to the number of documents. Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1YbHrfWT2LcQ%3DmQQY-Mdi\_m\_cJ%2BT%3Dq9%2B-ALStD5GtJ5g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1YbHrfWT2LcQ%3DmQQY-Mdi_m_cJ%2BT%3Dq9%2B-ALStD5GtJ5g%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [December 12, 2013, 9:26pm UTC](https://discuss.elastic.co/t/when-is-one-shard-enough/14840/2 "2013-12-12T21:26:53Z")
Why are you sharding on the first place? unless you know it to grow very big for sure, I'd skip this all together. Planning in advance to avoid possible resharding later can easily become over-engineering. Depending on usage, I'd go with having a shard up to dozen or 2 of GBs. That's an ideal size from my experience. If you want to go the virtual shard route nevertheless, try using a shard-key that will leave you with empty shards now and that you can bias later (for example - by using dates). HTH On Thu, Dec 12, 2013 at 6:13 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > I maintain search indexes for a bunch of wikis (hundreds), many of which > are small (hundreds of documents), some of which are pretty big (a few > million documents). I've noticed that having five shards per index for the > small wikis seems like overkill. It feels like the cluster is doing more > work maintaining index metadata for some wikis then it is for lucene > stuff. OTOH, my master node still has very low CPU usage. Does anyone run > with one or two shards for small indexes? > > Caveats: > > 1. Technically I split each wiki into two indexes to make the suggestion > results better. The split ends up putting about 3/4 of the documents in > one index and 1/4 in the other. In any case, that means even more shards > with even less data in them for the small wikis. > 2. Update and search rate is mostly proportional to the number of > documents. > > Nik > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1YbHrfWT2LcQ%3DmQQY-Mdi\_m\_cJ%2BT%3Dq9%2B-ALStD5GtJ5g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd1YbHrfWT2LcQ%3DmQQY-Mdi_m_cJ%2BT%3Dq9%2B-ALStD5GtJ5g%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuG278srFsVTrmifgyN%2B2wU7A6VkdXPeoO518-TXzZVvg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuG278srFsVTrmifgyN%2B2wU7A6VkdXPeoO518-TXzZVvg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 12, 2013, 10:20pm UTC](https://discuss.elastic.co/t/when-is-one-shard-enough/14840/3 "2013-12-12T22:20:09Z")
On Thu, Dec 12, 2013 at 4:26 PM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com)wrote: > Why are you sharding on the first place? unless you know it to grow very > big for sure, I'd skip this all together. Planning in advance to avoid > possible resharding later can easily become over-engineering. I'll take that as a yes. I'm not sure if I'd consider accepting the defaults to be over-engineering though. Depending on usage, I'd go with having a shard up to dozen or 2 of GBs. > That's an ideal size from my experience. That seems useful. Do you base that recommendation around ease of moving the shards around? By that logic the vast majority of the indexes I have now should be squished to one shard. None of them grow _that_ quickly and re-sharding them later really isn't a problem. If you want to go the virtual shard route nevertheless, try using a > shard-key that will leave you with empty shards now and that you can bias > later (for example - by using dates). I assume you mean custom routing. I can't really use that as there aren't any good route keys. None of the candidates break data into small enough buckets. In the case that you mean indexing documents based on time or some other key ala logstash, that kind of works for me and I already do it. I do it because it improves the output of the suggesters because each index has very different term frequencies. I don't get to delete any of the indexes over time, though, because they aren't time based. Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3CCcc4UUUe%2Bog-EMqh6DRi\_DD%2BDLDiGg2udNDX2BTz1w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3CCcc4UUUe%2Bog-EMqh6DRi_DD%2BDLDiGg2udNDX2BTz1w%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [December 12, 2013, 10:30pm UTC](https://discuss.elastic.co/t/when-is-one-shard-enough/14840/4 "2013-12-12T22:30:23Z")
inline On Fri, Dec 13, 2013 at 12:20 AM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > On Thu, Dec 12, 2013 at 4:26 PM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com)wrote: > > > Why are you sharding on the first place? unless you know it to grow very > > big for sure, I'd skip this all together. Planning in advance to avoid > > possible resharding later can easily become over-engineering. > > I'll take that as a yes. I'm not sure if I'd consider accepting the > defaults to be over-engineering though. I've heard Simon saying more than once that some of those defaults are actually pretty bad 🙂 > Depending on usage, I'd go with having a shard up to dozen or 2 of GBs. > > > That's an ideal size from my experience. > > That seems useful. Do you base that recommendation around ease of moving > the shards around? Mostly around what index size makes sense in the Lucene level, but obviously that depends on your analyzers etc. Moving shards around starts getting too costly to do at around 15GB. We've had a cluster that took days to stabilize when it got unstable because of larger indexes. > By that logic the vast majority of the indexes I have now should be > squished to one shard. None of them grow _that_ quickly and re-sharding > them later really isn't a problem. > > If you want to go the virtual shard route nevertheless, try using a > > > shard-key that will leave you with empty shards now and that you can bias > > later (for example - by using dates). > > I assume you mean custom routing. I can't really use that as there aren't > any good route keys. None of the candidates break data into small enough > buckets. > > In the case that you mean indexing documents based on time or some other > key ala logstash, that kind of works for me and I already do it. I do it > because it improves the output of the suggesters because each index has > very different term frequencies. I don't get to delete any of the indexes > over time, though, because they aren't time based. I meant routing. Unless you really handle lots of data, time sliced indexing is an alternative to sharding in most cases. Combining both (instead of using aliases, for example) only makes sense if you have to use some predefined slice size (like daily) and you have tons of data coming in. > Nik > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3CCcc4UUUe%2Bog-EMqh6DRi\_DD%2BDLDiGg2udNDX2BTz1w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3CCcc4UUUe%2Bog-EMqh6DRi_DD%2BDLDiGg2udNDX2BTz1w%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZujjpOjSxwb%2B9miwKw6Sic-OuHMMW4sXvrG80L24UK59w%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZujjpOjSxwb%2B9miwKw6Sic-OuHMMW4sXvrG80L24UK59w%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 13, 2013, 5:26pm UTC](https://discuss.elastic.co/t/when-is-one-shard-enough/14840/5 "2013-12-13T17:26:48Z")
inline On Thu, Dec 12, 2013 at 5:30 PM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com)wrote: > On Fri, Dec 13, 2013 at 12:20 AM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com)wrote: > > > On Thu, Dec 12, 2013 at 4:26 PM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com)wrote: > > > Depending on usage, I'd go with having a shard up to dozen or 2 of GBs. > > > > > That's an ideal size from my experience. > > > > That seems useful. Do you base that recommendation around ease of moving > > the shards around? > > Mostly around what index size makes sense in the Lucene level, but > obviously that depends on your analyzers etc. Moving shards around starts > getting too costly to do at around 15GB. We've had a cluster that took days > to stabilize when it got unstable because of larger indexes. Neat! Well, most wikis are getting squished to one shard then. What kind of thing about my choice of analyzer would suggest needing larger or smaller shards? Do you have a sense as to how many shards is too many? I assume search and indexing would speed up until you get to more shards than you have nodes. After that increasing shards would slow down search and indexing but keep the shards to a more manageable size. I remember reading about people complaining about performance when their index spanned thousands of shards and they wanted to search them all. I won't get that big, but could imagine fifty or a hundred. Thanks! Nik -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0Sd3cx2pZ%2B%2B%3DwqASRc\_E%2BxmFZL3kqX0Zrb7HnRG06VVA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0Sd3cx2pZ%2B%2B%3DwqASRc_E%2BxmFZL3kqX0Zrb7HnRG06VVA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [December 14, 2013, 4:03pm UTC](https://discuss.elastic.co/t/when-is-one-shard-enough/14840/6 "2013-12-14T16:03:26Z")
inline On Fri, Dec 13, 2013 at 7:26 PM, Nikolas Everett [nik9000@gmail.com](mailto:nik9000@gmail.com) wrote: > > Mostly around what index size makes sense in the Lucene level, but > > obviously that depends on your analyzers etc. Moving shards around starts > > getting too costly to do at around 15GB. We've had a cluster that took days > > to stabilize when it got unstable because of larger indexes. > > Neat! Well, most wikis are getting squished to one shard then. What kind > of thing about my choice of analyzer would suggest needing larger or > smaller shards? We use custom analyzers with stemming which double the amount of terms (multiple terms on the same position), and some even more than that. Think SynonymFilter and similar. Or if you index the same data multiple times. This bloats the index. > Do you have a sense as to how many shards is too many? I assume search > and indexing would speed up until you get to more shards than you have > nodes. After that increasing shards would slow down search and indexing > but keep the shards to a more manageable size. I remember reading about > people complaining about performance when their index spanned thousands of > shards and they wanted to search them all. I won't get that big, but could > imagine fifty or a hundred. Not really, and I believe this is tightly coupled with your data and expected set of queries. The ultimate idea behind sharding is to be able to make it so a query can execute on some of the shards to get full results, because the other shards return 0 results. The sharding key / function would have to try and do that. FWIW I have indexed the English wikipedia on my MacBook Air and it takes about 28GB. I can probably reduced more since I didn't use stemming nor ASCIIFoldingFilter. Its perfectly fine to have this as one shard assuming the growth rate is moderate. Just to be on the safe side you can shard it to 3 shards. To the best of my knowledge thats the largest Wiki there is, so thats your worst case scenario. > Thanks! > > Nik > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0Sd3cx2pZ%2B%2B%3DwqASRc\_E%2BxmFZL3kqX0Zrb7HnRG06VVA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0Sd3cx2pZ%2B%2B%3DwqASRc_E%2BxmFZL3kqX0Zrb7HnRG06VVA%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zte4Yn7SOjvOpSw2aZ3RHJjdiZnbysqqXO%2BVL0sJwUp5g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zte4Yn7SOjvOpSw2aZ3RHJjdiZnbysqqXO%2BVL0sJwUp5g%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:01am UTC](https://discuss.elastic.co/t/when-is-one-shard-enough/14840/7 "2017-07-06T02:01:12Z")