# Shards Initializing Indefinitely? **URL:** \ **Category:** Elasticsearch\ **Created:** [September 20, 2017, 3:42pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204 "2017-09-20T15:42:38Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![corona](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@corona](https://discuss.elastic.co/u/corona)\ **Post date:** [September 20, 2017, 3:42pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/1 "2017-09-20T15:42:38Z")
Hello, We are currently running version 2.4.0 in our production cluster. After node restarts I've noticed we have around 36 shards which seem to be stuck initializing. We have been having problems with nodes crashing due to OOM errors. Overtime (and many node restarts later: -- This is separate issue I'm trying to address) we have ended up with a bunch of initializing shards which never seem to finish initializing. I'll update this with more info/logs when they become available. I was wondering if there is anything I can do to address this issue other than delete the index / restart the node? At this point the cluster is red. In this state, can this have a slowing impact on overall performance? Could this slow down indexing b/c the cluster is also busying trying to initialize shards? Is there a way to determine if the shard has become corrupted and just won't ever initialize? Running the cat/\_recovery api tells me the following: index shard time type stage source\_host target\_host repository snapshot files files\_percent bytes bytes\_percent total\_files total\_bytes translog translog\_percent total\_translog xyz-index 0 596067535 store init 10.10.10.10 10.10.10.10 n/a n/a 0 0.0% 0 0.0% 0 0 0 -1.0% -1 Prior to this index/shard not initializing I verified it contained 57 docs and was about 23KB so I'm thinking it should have initialized pretty quickly. Any thoughts are greatly appreciated! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 20, 2017, 3:55pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/2 "2017-09-20T15:55:49Z")
How many shards do you have? How many nodes? ---
**Author:** ![corona](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@corona](https://discuss.elastic.co/u/corona)\ **Post date:** [September 20, 2017, 4:09pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/3 "2017-09-20T16:09:47Z")
We're running a 20 node cluster with ~48600 shards. { "cluster\_name" : "testxyz", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 20, "number\_of\_data\_nodes" : 20, "active\_primary\_shards" : 48621, "active\_shards" : 48621, "relocating\_shards" : 0, "initializing\_shards" : 36, "unassigned\_shards" : 0, "delayed\_unassigned\_shards" : 0, "number\_of\_pending\_tasks" : 4, "number\_of\_in\_flight\_fetch" : 0, "task\_max\_waiting\_in\_queue\_millis" : 7893, "active\_shards\_percent\_as\_number" : 99.92601270115297 } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 20, 2017, 4:16pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/4 "2017-09-20T16:16:18Z")
I think you have too many shards, and possibly also too many indices. The fact that you do not seem to use any replica shards probably does not help either. Would recommend you read [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 21, 2017, 12:40am UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/5 "2017-09-21T00:40:03Z")
Please don't add replicas on this cluster though. You are already _ **way** _ over sharded and you need to reduce that first before you start adding replicas in. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 21, 2017, 9:25am UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/6 "2017-09-21T09:25:22Z")
How much data do you have in the cluster? ---
**Author:** ![corona](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@corona](https://discuss.elastic.co/u/corona)\ **Post date:** [September 25, 2017, 3:30pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/7 "2017-09-25T15:30:39Z")
We have approximately ~17TB of data in our cluster. I was looking at a 6 month sample of data index breakdowns for our cluster. I'm seeing 90% of the indexes are pretty small in size. They can range anywhere from 50KB up to 48MB in size. We also have 3 groups of indices which make up the most of the data. They can range from 11-14GB in size. For the smaller indices, they are time based broken up by hour. The group of larger indexes are also time based but broken up by days. It sounds like recommendation is to decrease the number of shards in order to collapse the smaller indices into larger indices. Do you have a recommendation for doing that with version 2 of ES? Would we just need to reindex the data? Any thoughts are appreciated! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 25, 2017, 3:34pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/8 "2017-09-25T15:34:28Z")
While you can change settings for new indices getting created, you will indeed need to reindex older indices in order to reduce shard count. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 25, 2017, 10:36pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/9 "2017-09-25T22:36:45Z")
Unless you upgrade to 5.X (which you should totally do) and use the `_shrink` API 🙂 ---
**Author:** ![corona](https://avatars.discourse-cdn.com/v4/letter/c/958977/32.png) [@corona](https://discuss.elastic.co/u/corona)\ **Post date:** [September 26, 2017, 2:40pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/10 "2017-09-26T14:40:32Z")
Hey thanks for all of the quick responses from everyone. Very much appreciated!. Ok, let me know if you want me to open a new topic since this a different question, but I was curious about something related to this cluster. Recently I updated the bootstrap.mlock setting from false to true for half of the nodes (10). After about 1-2 weeks later, I'm seeing the nodes (with that setting enabled) are starting to use some swap space as it's slowly climbing. -- If the cluster is under a lot of stress due to too many indices/shards (as we described above), is it possible this setting will be ignored, or do you think there something else going on that would cause this setting to be ignored. If I understand correctly, enabling this setting will prevent the ES processes from swapping correct? I'm a little confused because I'm seeing something different. Again I realize our cluster is not in an ideal state. Thanks again. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 24, 2017, 2:40pm UTC](https://discuss.elastic.co/t/shards-initializing-indefinitely/101204/11 "2017-10-24T14:40:33Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.