# Replica shard stuck at initializing after client and data node restart **URL:** \ **Category:** Elasticsearch\ **Created:** [March 21, 2014, 7:15pm UTC](https://discuss.elastic.co/t/replica-shard-stuck-at-initializing-after-client-and-data-node-restart/16529 "2014-03-21T19:15:44Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Glenn\_Snead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glenn_snead/32/1699_2.png) [@Glenn\_Snead](https://discuss.elastic.co/u/Glenn_Snead)\ **Post date:** [March 21, 2014, 7:15pm UTC](https://discuss.elastic.co/t/replica-shard-stuck-at-initializing-after-client-and-data-node-restart/16529/1 "2014-03-21T19:15:44Z")
I have a six node cluster: 2 master nodes and 4 client / data nodes. I have two indicies. One with data and one that is set aside for future use. I'm having trouble with the indicie that is in use. After making some limits.conf configuraiton changes and restarting the impacted nodes, one of my indicies' replica shard will not complete initialization. I wasn't in charge of the node restarts and here is the sequence of events: Shut down the client and data nodes on each of the four servers. Start the client and data node on each server. I don't believe time was allowed to allow the cluster to reallocate or to move shards. limits.conf changes: - memlock unlimited hard nofiles 32000 soft nofiles 32000 Here's what I have tried thus far: Drop the replica shard, which brings the cluster status to Green. Verify the cluster's status - no replication, no realocating, etc. Re-add the replica shard. Drop the replica shard and the data nodes that were to carry the replica shard. Verify the cluster's status. Start the data nodes and allow the cluster to reallocate primary shards. - The cluster's status is Green. Add the replica shard to the indicie. The replica shard never completes initialization, even over a 24 hour period. I've checked the transaction log files on each node and they are all zero legnth files. The replica shard holding nodes are primary shards for the unused indicie. These nodes copied it's matching primary node's index Size (as seen in paramedic), but now Paramedic shows an index Size of only a few bytes. The index folder on the replica shard servers still has the data. Unknown to me, my target system was put online and my leadership doesn't want to schedule an outage window. Most my reasearch suggests that I drop the impacted indicie and re-initialize. I can replace the data, but this would impact the user interface while the indicie re-ingests the documents. This issue has occured before on my test system and the fix was to rebuild the index. However I never learned why the replica shard had the issue in the first place. My questions are: - Does the replica shard hosting server's index Size (shown in paramedic) indciate a course of action? - Is it possible to resolve this without dropping the indicie and rebuilding? I'd hate to resort to this each time we attempt ES server maintenance or configuration changes. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [March 22, 2014, 1:48am UTC](https://discuss.elastic.co/t/replica-shard-stuck-at-initializing-after-client-and-data-node-restart/16529/2 "2014-03-22T01:48:27Z")
What version are you running? It's odd this would happen if, when you set replica's to zero, the cluster state is green and your index is ok. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 22 March 2014 06:15, Glenn Snead [glennsnead@gmail.com](mailto:glennsnead@gmail.com) wrote: > I have a six node cluster: 2 master nodes and 4 client / data nodes. I > have two indicies. One with data and one that is set aside for future > use. I'm having trouble with the indicie that is in use. > After making some limits.conf configuraiton changes and restarting the > impacted nodes, one of my indicies' replica shard will not complete > initialization. > I wasn't in charge of the node restarts and here is the sequence of events: > Shut down the client and data nodes on each of the four servers. > Start the client and data node on each server. > I don't believe time was allowed to allow the cluster to reallocate or to > move shards. > > limits.conf changes: > > - memlock unlimited > hard nofiles 32000 > soft nofiles 32000 > > Here's what I have tried thus far: > > Drop the replica shard, which brings the cluster status to Green. > Verify the cluster's status - no replication, no realocating, etc. > Re-add the replica shard. > > Drop the replica shard and the data nodes that were to carry the replica > shard. > Verify the cluster's status. > Start the data nodes and allow the cluster to reallocate primary shards. > > - The cluster's status is Green. > Add the replica shard to the indicie. The replica shard never completes > initialization, even over a 24 hour period. > > I've checked the transaction log files on each node and they are all zero > legnth files. > The replica shard holding nodes are primary shards for the unused indicie. > These nodes copied it's matching primary node's index Size (as seen in > paramedic), but now Paramedic shows an index Size of only a few bytes. The > index folder on the replica shard servers still has the data. > > Unknown to me, my target system was put online and my leadership doesn't > want to schedule an outage window. Most my reasearch suggests that I drop > the impacted indicie and re-initialize. I can replace the data, but this > would impact the user interface while the indicie re-ingests the > documents. This issue has occured before on my test system and the fix was > to rebuild the index. However I never learned why the replica shard had > the issue in the first place. > > My questions are: > > - Does the replica shard hosting server's index Size (shown in paramedic) > indciate a course of action? > - Is it possible to resolve this without dropping the indicie and > rebuilding? I'd hate to resort to this each time we attempt ES server > maintenance or configuration changes. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ZsMpqnF16T\_3-ZzDy7SjcsFouaDOBQQEEATby%2B7Lorzg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ZsMpqnF16T_3-ZzDy7SjcsFouaDOBQQEEATby%2B7Lorzg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Glenn\_Snead](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/glenn_snead/32/1699_2.png) [@Glenn\_Snead](https://discuss.elastic.co/u/Glenn_Snead)\ **Post date:** [March 24, 2014, 3:20pm UTC](https://discuss.elastic.co/t/replica-shard-stuck-at-initializing-after-client-and-data-node-restart/16529/3 "2014-03-24T15:20:07Z")
Mark, thank you for responding. It is quite odd, but what happened this morning is stranger. I dropped and recreated the replica shard on the unused indicie. Now the in-use indicie shows Green. FYI we're running ES version 0.90, and my in-use indicie is 717 gb with 135M+ documents. On Friday I ran status reports on each indicie and compared both. Nothing showed as "failed" or "red" or plain "wrong" so I left it over the weekend. When I came in today the cluster was still Yellow. Any idea if createating the other indicie's replica shard caused the cluster's status to go green? It feels like a fluke, but I'm new to ES. If this is indeed an expected ES behavior, I'll add this to my restoral procedures. On Friday, March 21, 2014 9:48:27 PM UTC-4, Mark Walkom wrote: > What version are you running? > > It's odd this would happen if, when you set replica's to zero, the cluster > state is green and your index is ok. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 22 March 2014 06:15, Glenn Snead \<[glenn...@gmail.com](mailto:glenn...@gmail.com) \\>wrote: > > > I have a six node cluster: 2 master nodes and 4 client / data nodes. I > > have two indicies. One with data and one that is set aside for future > > use. I'm having trouble with the indicie that is in use. > > After making some limits.conf configuraiton changes and restarting the > > impacted nodes, one of my indicies' replica shard will not complete > > initialization. > > I wasn't in charge of the node restarts and here is the sequence of > > events: > > Shut down the client and data nodes on each of the four servers. > > Start the client and data node on each server. > > I don't believe time was allowed to allow the cluster to reallocate or to > > move shards. > > > > limits.conf changes: > > > > - memlock unlimited > > hard nofiles 32000 > > soft nofiles 32000 > > > > Here's what I have tried thus far: > > > > Drop the replica shard, which brings the cluster status to Green. > > Verify the cluster's status - no replication, no realocating, etc. > > Re-add the replica shard. > > > > Drop the replica shard and the data nodes that were to carry the replica > > shard. > > Verify the cluster's status. > > Start the data nodes and allow the cluster to reallocate primary shards. > > > > - The cluster's status is Green. > > Add the replica shard to the indicie. The replica shard never completes > > initialization, even over a 24 hour period. > > > > I've checked the transaction log files on each node and they are all zero > > legnth files. > > The replica shard holding nodes are primary shards for the unused indicie. > > These nodes copied it's matching primary node's index Size (as seen in > > paramedic), but now Paramedic shows an index Size of only a few bytes. The > > index folder on the replica shard servers still has the data. > > > > Unknown to me, my target system was put online and my leadership doesn't > > want to schedule an outage window. Most my reasearch suggests that I drop > > the impacted indicie and re-initialize. I can replace the data, but this > > would impact the user interface while the indicie re-ingests the > > documents. This issue has occured before on my test system and the fix was > > to rebuild the index. However I never learned why the replica shard had > > the issue in the first place. > > > > My questions are: > > > > - Does the replica shard hosting server's index Size (shown in paramedic) > > indciate a course of action? > > - Is it possible to resolve this without dropping the indicie and > > rebuilding? I'd hate to resort to this each time we attempt ES server > > maintenance or configuration changes. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/a868b3da-fd28-49b4-bc8f-2f60f2c34ec7%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ed2501e5-b504-46e2-ae04-69097e6d46ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ed2501e5-b504-46e2-ae04-69097e6d46ed%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:41am UTC](https://discuss.elastic.co/t/replica-shard-stuck-at-initializing-after-client-and-data-node-restart/16529/4 "2017-07-06T01:41:10Z")