# Downsides of over-allocating shards? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 25, 2012, 7:12am UTC](https://discuss.elastic.co/t/downsides-of-over-allocating-shards/6488 "2012-01-25T07:12:33Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [January 25, 2012, 7:12am UTC](https://discuss.elastic.co/t/downsides-of-over-allocating-shards/6488/1 "2012-01-25T07:12:33Z")
Hi, Shard over-allocation was nicely covered in some recent threads. Assuming one uses routing and filtering to limit queries to a specific shard, can one over-allocate shards to the point of causing problems? For example: Say that you determine that having 20 shards in some cluster would work well right now, but you want to plan for growth over the next 12 months, so you opt to go with 100 shards. But what if you are super certain about growth or simply want to plan for more than the next 12 months. Why not just choose, say, 1000 shards? Is there anything that can go wrong? Any negative side effects one should be aware or? Thanks, Otis ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 25, 2012, 4:32pm UTC](https://discuss.elastic.co/t/downsides-of-over-allocating-shards/6488/2 "2012-01-25T16:32:03Z")
Only the overhead a shard has, which is a full Lucene index (memory, file descriptors). It really depends on the HW you have and what you do with it. On Wednesday, January 25, 2012 at 9:12 AM, Otis Gospodnetic wrote: > Hi, > > Shard over-allocation was nicely covered in some recent threads. > Assuming one uses routing and filtering to limit queries to a specific > shard, can one over-allocate shards to the point of causing problems? > > For example: > > Say that you determine that having 20 shards in some cluster would > work well right now, but you want to plan for growth over the next 12 > months, so you opt to go with 100 shards. > But what if you are super certain about growth or simply want to plan > for more than the next 12 months. Why not just choose, say, 1000 > shards? > > Is there anything that can go wrong? Any negative side effects one > should be aware or? > > Thanks, > Otis ---
**Author:** ![phobos182](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phobos182/32/3011_2.png) [@phobos182](https://discuss.elastic.co/u/phobos182)\ **Post date:** [January 29, 2012, 1:47pm UTC](https://discuss.elastic.co/t/downsides-of-over-allocating-shards/6488/3 "2012-01-29T13:47:05Z")
We are in the process of reversing a decision we made along these times. The duplication of the TermInfoIndex had a pretty good size memory cost. Reducing the number of shards reduced memory pressure on the nodes so more memory was available for caching. IMHO there is a pretty good size memory cost loading the index data into memory. So we try to use a few shards as possible to have search results return in an acceptable time. ---
**Author:** ![phobos182](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phobos182/32/3011_2.png) [@phobos182](https://discuss.elastic.co/u/phobos182)\ **Post date:** [February 1, 2012, 6:27pm UTC](https://discuss.elastic.co/t/downsides-of-over-allocating-shards/6488/4 "2012-02-01T18:27:49Z")
One issue we recently ran into Shay. I wanted to get your feedback. In our development environment we went 100% the other way, and decreased the number of indexes, shards, and increased our tiered merging policy from 5g to 20gb to reduce segments in the system. This resulted an increase in segments size, and de-duplicated a lot of the TermInfoIndex items. After doing this our memory looked great! We did not have to touch the TermIndex Divisor, or anything like that. As an unintended side effect, search / query time went through the roof. Looking at some stack traces, a LOT of time was being spent performing the searches themselves Example: org.apache.lucene.index.SegmentTermPositions.lazySkip(SegmentTermPositions.java:169) It looks like a single threaded operations (1 CPU). So my question is this. Do we increase the number of shards to increase CPU concurrency thus decreasing search time and making it more performant? For example: ## 2 servers. 12 CPU's each. 1 Index: 24 shards evenly balanced. A search operation will optimistically (if allocated fair) use all 24 CPU's in the system. The cost in RAM increases quite a bit as TermInfo's are duplicated, more segments, more stuff to load into memory. 1 Index: 2 shards evenly balanced. A search operation will use 2 CPU's in the system. RAM is reduced quite a bit, segments decrease, etc.. Is this correct? Thanks! ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [February 2, 2012, 5:36am UTC](https://discuss.elastic.co/t/downsides-of-over-allocating-shards/6488/5 "2012-02-02T05:36:04Z")
Hi, On Feb 1, 1:27 pm, phobos182 [phobos...@gmail.com](mailto:phobos...@gmail.com) wrote: > One issue we recently ran into Shay. I wanted to get your feedback. In our > development environment we went 100% the other way, and decreased the number > of indexes, shards, and increased our tiered merging policy from 5g to 20gb > to reduce segments in the system. This resulted an increase in segments > size, and de-duplicated a lot of the TermInfoIndex items. > > After doing this our memory looked great! We did not have to touch the > TermIndex Divisor, or anything like that. As an unintended side effect, > search / query time went through the roof. Looking at some stack traces, a > LOT of time was being spent performing the searches themselves Example: > > org.apache.lucene.index.SegmentTermPositions.lazySkip(SegmentTermPositions. java:169) > > It looks like a single threaded operations (1 CPU). So my question is this. > Do we increase the number of shards to increase CPU concurrency thus > decreasing search time and making it more performant? For example: Yes. 🙂 It's faster for a 12 core box to search 12 smaller indices in parallel than 1 large index. The total time for the former will be about 1/12 of the latter. > ## 2 servers. 12 CPU's each. > > 1 Index: 24 shards evenly balanced. > A search operation will optimistically (if allocated fair) use all 24 CPU's > in the system. The cost in RAM increases quite a bit as TermInfo's are > duplicated, more segments, more stuff to load into memory. > > 1 Index: 2 shards evenly balanced. > A search operation will use 2 CPU's in the system. RAM is reduced quite a > bit, segments decrease, etc.. > > Is this correct? Yes, I think so. ## Otis > **[Sematext | IT System Monitoring Tools for DevOps](https://sematext.com/)** > > IT system monitoring and management tools for DevOps who need 24x7 live visibility into their infrastructure. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:40am UTC](https://discuss.elastic.co/t/downsides-of-over-allocating-shards/6488/6 "2017-07-06T03:40:44Z")