# Very weird ES Cluster state problem! **URL:** \ **Category:** Elasticsearch\ **Created:** [February 1, 2013, 6:01pm UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594 "2013-02-01T18:01:07Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Amit](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@Amit](https://discuss.elastic.co/u/Amit)\ **Post date:** [February 1, 2013, 6:01pm UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594/1 "2013-02-01T18:01:07Z")
Hi All, I am having an ES cluster with 2 nodes. I am not as to what caused this issue; Node 2- [2] received shard failed for [TestDocTestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local recovery (post api), should exists, but doesn't]]] [2013-02-01 16:37:41,667][WARN][indices.cluster] [2] [TestDoc][2] failed to start shard org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [TestDoc][2] shard allocated for local recovery (post api), should exists, but doesn't at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:108) at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:636) [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] sending failed shard for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local recovery (post api), should exists, but doesn't]]] [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] received shard failed for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local recovery (post api), should exists, but doesn't]]] [2013-02-01 16:39:00,306][WARN][discovery.zen] [2] master should not receive new cluster state from [[1][IlJPr1CBTmKxSgHyHJ7brg][inet[/10.190.209.134:9300]]] Node1- [2013-02-01 10:08:03,861][DEBUG][action.search.type] [1] failed to reduce search org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase [fetch], [reduce] at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:177) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:155) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:1) at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:345) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:149) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:136) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: java.lang.ClassCastException: org.elasticsearch.search.facet.termsstats.longs.InternalTermsStatsLongFacet cannot be cast to org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti at org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti.reduce(InternalTermsStatsStringFacetMulti.java:490) at org.elasticsearch.plugin.multifssearch.TermsStatsFacetProcessorMulti.reduce(TermsStatsFacetProcessorMulti.java:166) at org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:296) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:190) at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:175) ... 8 more [2013-02-01 12:59:42,516][INFO][cluster.metadata] [1] [TestDoc2] creating index, cause [auto(bulk api)], shards [5]/[0], mappings [TestDoc2~type1] [2013-02-01 13:00:34,555][INFO][cluster.metadata] [1] [TestDoc3] creating index, cause [auto(bulk api)], shards [5]/[0], mappings [TestDoc3~type1] The Cluster health api from node1; { "cluster\_name" : "test1", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 1, "number\_of\_data\_nodes" : 1, "active\_primary\_shards" : 4256, "active\_shards" : 4256, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 4209 } The Cluster health api from node2; { "cluster\_name" : "test", "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 2, "number\_of\_data\_nodes" : 2, "active\_primary\_shards" : 8471, "active\_shards" : 8471, "relocating\_shards" : 0, "initializing\_shards" : 4, "unassigned\_shards" : 0 } I looked through the ES group but could not find the exact issue. It looks like one of the node ( primary) left the cluster because of the network issue( not sure what was the issue, assuming network issue). And the secondary got elected as master. And when the network issue was resolved. The primary node was trying to join the cluster, which did happen. But probably the state was not synched? or there two master nodes master1- having two node in cluster, but not able to communicate with data node. master2- having only one node in cluster. Please help me as this is going crazy over my head. I looked through the different threads, but nothing concrete. Thanks in advance Amit -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [February 1, 2013, 10:18pm UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594/2 "2013-02-01T22:18:35Z")
Yes, it looks like a crash or a split. The problematic indexes are lost, and should be deleted. Otherwise, ES is not able to resolve the conflict. Note, there are precautions against such node splits, did you change the default settings in zen discovery, minimum\_master\_nodes for example? Best regards, Jörg Am 01.02.13 19:01, schrieb Amit Singh: > Hi All, > > I am having an ES cluster with 2 nodes. I am not as to what caused > this issue; > > Node 2- > > [2] received shard failed for [TestDocTestDoc][2], > node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to > start shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] > shard allocated for local recovery (post api), should exists, but > doesn't]]] > [2013-02-01 16:37:41,667][WARN][indices.cluster] [2] > [TestDoc][2] failed to start shard > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [TestDoc][2] shard allocated for local recovery (post api), should > exists, but doesn't > at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:108) > at > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:636) > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > sending failed shard for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], > [P], s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for > local recovery (post api), should exists, but doesn't]]] > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > received shard failed for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], > [P], s[INITIALIZING], reason [Failed to start shard, message > [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for > local recovery (post api), should exists, but doesn't]]] > [2013-02-01 16:39:00,306][WARN][discovery.zen] [2] master > should not receive new cluster state from > [[1][IlJPr1CBTmKxSgHyHJ7brg][inet[/10.190.209.134:9300]]] > > Node1- > > [2013-02-01 10:08:03,861][DEBUG][action.search.type] [1] failed > to reduce search > org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to > execute phase [fetch], [reduce] > at > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:177) > at > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:155) > at > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:1) > at > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:345) > at > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:149) > at > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:136) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.lang.ClassCastException: > org.elasticsearch.search.facet.termsstats.longs.InternalTermsStatsLongFacet > cannot be cast to > org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti > at > org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti.reduce(InternalTermsStatsStringFacetMulti.java:490) > at > org.elasticsearch.plugin.multifssearch.TermsStatsFacetProcessorMulti.reduce(TermsStatsFacetProcessorMulti.java:166) > at > org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:296) > at > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:190) > at > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:175) > ... 8 more > [2013-02-01 12:59:42,516][INFO][cluster.metadata] [1] > [TestDoc2] creating index, cause [auto(bulk api)], shards [5]/[0], > mappings [TestDoc2~type1] > [2013-02-01 13:00:34,555][INFO][cluster.metadata] [1] > [TestDoc3] creating index, cause [auto(bulk api)], shards [5]/[0], > mappings [TestDoc3~type1] > > The Cluster health api from node1; > > { > "cluster\_name" : "test1", > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 1, > "number\_of\_data\_nodes" : 1, > "active\_primary\_shards" : 4256, > "active\_shards" : 4256, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 4209 > } > > The Cluster health api from node2; > { > "cluster\_name" : "test", > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 2, > "number\_of\_data\_nodes" : 2, > "active\_primary\_shards" : 8471, > "active\_shards" : 8471, > "relocating\_shards" : 0, > "initializing\_shards" : 4, > "unassigned\_shards" : 0 > } > > I looked through the ES group but could not find the exact issue. > It looks like one of the node ( primary) left the cluster because of > the network issue( not sure what was the issue, assuming network > issue). And the secondary got elected as master. And when the network > issue was resolved. The primary node was trying to join the cluster, > which did happen. But probably the state was not synched? or there two > master nodes master1- having two node in cluster, but not able to > communicate with data node. master2- having only one node in cluster. > > Please help me as this is going crazy over my head. I looked through > the different threads, but nothing concrete. > > ## Thanks in advance Amit > > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 2, 2013, 10:12pm UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594/3 "2013-02-02T22:12:02Z")
Jorg mentioned the important minimum\_master\_nodes, which version of ES are you on? On Feb 1, 2013, at 11:18 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Yes, it looks like a crash or a split. The problematic indexes are lost, and should be deleted. Otherwise, ES is not able to resolve the conflict. Note, there are precautions against such node splits, did you change the default settings in zen discovery, minimum\_master\_nodes for example? > > Best regards, > > Jörg > > Am 01.02.13 19:01, schrieb Amit Singh: > > > Hi All, > > > > I am having an ES cluster with 2 nodes. I am not as to what caused this issue; > > > > Node 2- > > > > [2] received shard failed for [TestDocTestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local recovery (post api), should exists, but doesn't]]] > > [2013-02-01 16:37:41,667][WARN][indices.cluster] [2] [TestDoc][2] failed to start shard > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [TestDoc][2] shard allocated for local recovery (post api), should exists, but doesn't > > at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:108) > > at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > at java.lang.Thread.run(Thread.java:636) > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] sending failed shard for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local recovery (post api), should exists, but doesn't]]] > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] received shard failed for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local recovery (post api), should exists, but doesn't]]] > > [2013-02-01 16:39:00,306][WARN][discovery.zen] [2] master should not receive new cluster state from [[1][IlJPr1CBTmKxSgHyHJ7brg][inet[/10.190.209.134:9300]]] > > > > Node1- > > > > [2013-02-01 10:08:03,861][DEBUG][action.search.type] [1] failed to reduce search > > org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to execute phase [fetch], [reduce] > > at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:177) > > at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:155) > > at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:1) > > at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:345) > > at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:149) > > at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:136) > > at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > at java.lang.Thread.run(Thread.java:662) > > Caused by: java.lang.ClassCastException: org.elasticsearch.search.facet.termsstats.longs.InternalTermsStatsLongFacet cannot be cast to org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti > > at org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti.reduce(InternalTermsStatsStringFacetMulti.java:490) > > at org.elasticsearch.plugin.multifssearch.TermsStatsFacetProcessorMulti.reduce(TermsStatsFacetProcessorMulti.java:166) > > at org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:296) > > at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:190) > > at org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:175) > > ... 8 more > > [2013-02-01 12:59:42,516][INFO][cluster.metadata] [1] [TestDoc2] creating index, cause [auto(bulk api)], shards [5]/[0], mappings [TestDoc2~type1] > > [2013-02-01 13:00:34,555][INFO][cluster.metadata] [1] [TestDoc3] creating index, cause [auto(bulk api)], shards [5]/[0], mappings [TestDoc3~type1] > > > > The Cluster health api from node1; > > > > { > > "cluster\_name" : "test1", > > "status" : "red", > > "timed\_out" : false, > > "number\_of\_nodes" : 1, > > "number\_of\_data\_nodes" : 1, > > "active\_primary\_shards" : 4256, > > "active\_shards" : 4256, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 4209 > > } > > > > The Cluster health api from node2; > > { > > "cluster\_name" : "test", > > "status" : "red", > > "timed\_out" : false, > > "number\_of\_nodes" : 2, > > "number\_of\_data\_nodes" : 2, > > "active\_primary\_shards" : 8471, > > "active\_shards" : 8471, > > "relocating\_shards" : 0, > > "initializing\_shards" : 4, > > "unassigned\_shards" : 0 > > } > > > > I looked through the ES group but could not find the exact issue. > > It looks like one of the node ( primary) left the cluster because of the network issue( not sure what was the issue, assuming network issue). And the secondary got elected as master. And when the network issue was resolved. The primary node was trying to join the cluster, which did happen. But probably the state was not synched? or there two master nodes master1- having two node in cluster, but not able to communicate with data node. master2- having only one node in cluster. > > > > Please help me as this is going crazy over my head. I looked through the different threads, but nothing concrete. > > > > ## Thanks in advance Amit > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Amit](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@Amit](https://discuss.elastic.co/u/Amit)\ **Post date:** [February 3, 2013, 5:08am UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594/4 "2013-02-03T05:08:56Z")
Thanks Jorg and kimchy, I am using ES 0.19.4 version and minimum\_master\_nodes setting is default. Since I have only two nodes in the cluster, I did not change the minimum\_master\_nodes, as N/2+1 will give me value of 1 for two nodes. Further when I restarted the cluster I am still getting the same error! Please suggest. Thanks Amit On Sunday, February 3, 2013 3:42:02 AM UTC+5:30, kimchy wrote: > Jorg mentioned the important minimum\_master\_nodes, which version of ES are > you on? > > On Feb 1, 2013, at 11:18 PM, Jörg Prante \<[joerg...@gmail.com](mailto:joerg...@gmail.com) \\> > wrote: > > > Yes, it looks like a crash or a split. The problematic indexes are lost, > > and should be deleted. Otherwise, ES is not able to resolve the conflict. > > Note, there are precautions against such node splits, did you change the > > default settings in zen discovery, minimum\_master\_nodes for example? > > > > Best regards, > > > > Jörg > > > > Am 01.02.13 19:01, schrieb Amit Singh: > > > > > Hi All, > > > > > > I am having an ES cluster with 2 nodes. I am not as to what caused this > > > issue; > > > > > > Node 2- > > > > > > [2] received shard failed for [TestDocTestDoc][2], > > > node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start > > > shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] shard > > > allocated for local recovery (post api), should exists, but doesn't]]] > > > [2013-02-01 16:37:41,667][WARN][indices.cluster] [2] > > > [TestDoc][2] failed to start shard > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > [TestDoc][2] shard allocated for local recovery (post api), should exists, > > > but doesn't > > > at > > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:108) > > > > at > > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > > > > at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > > > at java.lang.Thread.run(Thread.java:636) > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] sending > > > failed shard for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > s[INITIALIZING], reason [Failed to start shard, message > > > [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local > > > recovery (post api), should exists, but doesn't]]] > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > > > received shard failed for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > s[INITIALIZING], reason [Failed to start shard, message > > > [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local > > > recovery (post api), should exists, but doesn't]]] > > > [2013-02-01 16:39:00,306][WARN][discovery.zen] [2] master > > > should not receive new cluster state from > > > [[1][IlJPr1CBTmKxSgHyHJ7brg][inet[/10.190.209.134:9300]]] > > > > > > Node1- > > > > > > [2013-02-01 10:08:03,861][DEBUG][action.search.type] [1] failed > > > to reduce search > > > org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to > > > execute phase [fetch], [reduce] > > > at > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:177) > > > > at > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:155) > > > > at > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:1) > > > > at > > > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:345) > > > > at > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:149) > > > > at > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:136) > > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > at > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > at java.lang.Thread.run(Thread.java:662) > > > Caused by: java.lang.ClassCastException: > > > org.elasticsearch.search.facet.termsstats.longs.InternalTermsStatsLongFacet > > > cannot be cast to > > > org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti > > > at > > > org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti.reduce(InternalTermsStatsStringFacetMulti.java:490) > > > > at > > > org.elasticsearch.plugin.multifssearch.TermsStatsFacetProcessorMulti.reduce(TermsStatsFacetProcessorMulti.java:166) > > > > at > > > org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:296) > > > > at > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:190) > > > > at > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:175) > > > > ... 8 more > > > [2013-02-01 12:59:42,516][INFO][cluster.metadata] [1] > > > [TestDoc2] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > [TestDoc2~type1] > > > [2013-02-01 13:00:34,555][INFO][cluster.metadata] [1] > > > [TestDoc3] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > [TestDoc3~type1] > > > > > > The Cluster health api from node1; > > > > > > { > > > "cluster\_name" : "test1", > > > "status" : "red", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 1, > > > "number\_of\_data\_nodes" : 1, > > > "active\_primary\_shards" : 4256, > > > "active\_shards" : 4256, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 0, > > > "unassigned\_shards" : 4209 > > > } > > > > > > The Cluster health api from node2; > > > { > > > "cluster\_name" : "test", > > > "status" : "red", > > > "timed\_out" : false, > > > "number\_of\_nodes" : 2, > > > "number\_of\_data\_nodes" : 2, > > > "active\_primary\_shards" : 8471, > > > "active\_shards" : 8471, > > > "relocating\_shards" : 0, > > > "initializing\_shards" : 4, > > > "unassigned\_shards" : 0 > > > } > > > > > > I looked through the ES group but could not find the exact issue. > > > It looks like one of the node ( primary) left the cluster because of > > > the network issue( not sure what was the issue, assuming network issue). > > > And the secondary got elected as master. And when the network issue was > > > resolved. The primary node was trying to join the cluster, which did > > > happen. But probably the state was not synched? or there two master nodes > > > master1- having two node in cluster, but not able to communicate with data > > > node. master2- having only one node in cluster. > > > > > > Please help me as this is going crazy over my head. I looked through > > > the different threads, but nothing concrete. > > > > > > ## Thanks in advance Amit > > > > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Igor\_Motov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/igor_motov/32/45193_2.png) [@Igor\_Motov](https://discuss.elastic.co/u/Igor_Motov)\ **Post date:** [February 4, 2013, 6:59pm UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594/5 "2013-02-04T18:59:36Z")
Are these the actual responses or you changed the cluster names before posting them on the mailing list? The Cluster health api from node1; { "cluster\_name" : "_test1_", "status" : "red", The Cluster health api from node2; { "cluster\_name" : "_test_", "status" : "red", On Sunday, February 3, 2013 12:08:56 AM UTC-5, Amit Singh wrote: > Thanks Jorg and kimchy, > > I am using ES 0.19.4 version and minimum\_master\_nodes setting is default. > Since I have only two nodes in the cluster, I did not change > the minimum\_master\_nodes, as N/2+1 will give me value of 1 for two nodes. > Further when I restarted the cluster I am still getting the same error! > > Please suggest. > > Thanks > Amit > > On Sunday, February 3, 2013 3:42:02 AM UTC+5:30, kimchy wrote: > > > Jorg mentioned the important minimum\_master\_nodes, which version of ES > > are you on? > > > > On Feb 1, 2013, at 11:18 PM, Jörg Prante [joerg...@gmail.com](mailto:joerg...@gmail.com) wrote: > > > > > Yes, it looks like a crash or a split. The problematic indexes are > > > lost, and should be deleted. Otherwise, ES is not able to resolve the > > > conflict. Note, there are precautions against such node splits, did you > > > change the default settings in zen discovery, minimum\_master\_nodes for > > > example? > > > > > > Best regards, > > > > > > Jörg > > > > > > Am 01.02.13 19:01, schrieb Amit Singh: > > > > > > > Hi All, > > > > > > > > I am having an ES cluster with 2 nodes. I am not as to what caused > > > > this issue; > > > > > > > > Node 2- > > > > > > > > [2] received shard failed for [TestDocTestDoc][2], > > > > node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start > > > > shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] shard > > > > allocated for local recovery (post api), should exists, but doesn't]]] > > > > [2013-02-01 16:37:41,667][WARN][indices.cluster] [2] > > > > [TestDoc][2] failed to start shard > > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > > [TestDoc][2] shard allocated for local recovery (post api), should exists, > > > > but doesn't > > > > at > > > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:108) > > > > > > at > > > > org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177) > > > > > > at > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > > > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > > > > > > at java.lang.Thread.run(Thread.java:636) > > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > > > > sending failed shard for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > > s[INITIALIZING], reason [Failed to start shard, message > > > > [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local > > > > recovery (post api), should exists, but doesn't]]] > > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > > > > received shard failed for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > > s[INITIALIZING], reason [Failed to start shard, message > > > > [IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local > > > > recovery (post api), should exists, but doesn't]]] > > > > [2013-02-01 16:39:00,306][WARN][discovery.zen] [2] master > > > > should not receive new cluster state from > > > > [[1][IlJPr1CBTmKxSgHyHJ7brg][inet[/10.190.209.134:9300]]] > > > > > > > > Node1- > > > > > > > > [2013-02-01 10:08:03,861][DEBUG][action.search.type] [1] failed > > > > to reduce search > > > > org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to > > > > execute phase [fetch], [reduce] > > > > at > > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:177) > > > > > > at > > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:155) > > > > > > at > > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:1) > > > > > > at > > > > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:345) > > > > > > at > > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:149) > > > > > > at > > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:136) > > > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > > > > > at > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > > > > > at java.lang.Thread.run(Thread.java:662) > > > > Caused by: java.lang.ClassCastException: > > > > org.elasticsearch.search.facet.termsstats.longs.InternalTermsStatsLongFacet > > > > cannot be cast to > > > > org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti > > > > at > > > > org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti.reduce(InternalTermsStatsStringFacetMulti.java:490) > > > > > > at > > > > org.elasticsearch.plugin.multifssearch.TermsStatsFacetProcessorMulti.reduce(TermsStatsFacetProcessorMulti.java:166) > > > > > > at > > > > org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:296) > > > > > > at > > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:190) > > > > > > at > > > > org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:175) > > > > > > ... 8 more > > > > [2013-02-01 12:59:42,516][INFO][cluster.metadata] [1] > > > > [TestDoc2] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > > [TestDoc2~type1] > > > > [2013-02-01 13:00:34,555][INFO][cluster.metadata] [1] > > > > [TestDoc3] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > > [TestDoc3~type1] > > > > > > > > The Cluster health api from node1; > > > > > > > > { > > > > "cluster\_name" : "test1", > > > > "status" : "red", > > > > "timed\_out" : false, > > > > "number\_of\_nodes" : 1, > > > > "number\_of\_data\_nodes" : 1, > > > > "active\_primary\_shards" : 4256, > > > > "active\_shards" : 4256, > > > > "relocating\_shards" : 0, > > > > "initializing\_shards" : 0, > > > > "unassigned\_shards" : 4209 > > > > } > > > > > > > > The Cluster health api from node2; > > > > { > > > > "cluster\_name" : "test", > > > > "status" : "red", > > > > "timed\_out" : false, > > > > "number\_of\_nodes" : 2, > > > > "number\_of\_data\_nodes" : 2, > > > > "active\_primary\_shards" : 8471, > > > > "active\_shards" : 8471, > > > > "relocating\_shards" : 0, > > > > "initializing\_shards" : 4, > > > > "unassigned\_shards" : 0 > > > > } > > > > > > > > I looked through the ES group but could not find the exact issue. > > > > It looks like one of the node ( primary) left the cluster because of > > > > the network issue( not sure what was the issue, assuming network issue). > > > > And the secondary got elected as master. And when the network issue was > > > > resolved. The primary node was trying to join the cluster, which did > > > > happen. But probably the state was not synched? or there two master nodes > > > > master1- having two node in cluster, but not able to communicate with data > > > > node. master2- having only one node in cluster. > > > > > > > > Please help me as this is going crazy over my head. I looked through > > > > the different threads, but nothing concrete. > > > > > > > > ## Thanks in advance Amit > > > > > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [February 4, 2013, 7:26pm UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594/6 "2013-02-04T19:26:20Z")
Hi, For two nodes you should have minimum\_master\_nodes=2, otherwise you can end up with 2 1-node clusters. And 2/2+1=2 On Feb 3, 2013 7:09 AM, "Amit Singh" [amitsingh.kec@gmail.com](mailto:amitsingh.kec@gmail.com) wrote: > Thanks Jorg and kimchy, > > I am using ES 0.19.4 version and minimum\_master\_nodes setting is default. > Since I have only two nodes in the cluster, I did not change > the minimum\_master\_nodes, as N/2+1 will give me value of 1 for two nodes. > Further when I restarted the cluster I am still getting the same error! > > Please suggest. > > Thanks > Amit > > On Sunday, February 3, 2013 3:42:02 AM UTC+5:30, kimchy wrote: > > > Jorg mentioned the important minimum\_master\_nodes, which version of ES > > are you on? > > > > On Feb 1, 2013, at 11:18 PM, Jörg Prante [joerg...@gmail.com](mailto:joerg...@gmail.com) wrote: > > > > > Yes, it looks like a crash or a split. The problematic indexes are > > > lost, and should be deleted. Otherwise, ES is not able to resolve the > > > conflict. Note, there are precautions against such node splits, did you > > > change the default settings in zen discovery, minimum\_master\_nodes for > > > example? > > > > > > Best regards, > > > > > > Jörg > > > > > > Am 01.02.13 19:01, schrieb Amit Singh: > > > > > > > Hi All, > > > > > > > > I am having an ES cluster with 2 nodes. I am not as to what caused > > > > this issue; > > > > > > > > Node 2- > > > > > > > > [2] received shard failed for [TestDocTestDoc][2], > > > > node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start > > > > shard, message [**IndexShardGatewayRecoveryExcep** tion[[TestDoc][2] > > > > shard allocated for local recovery (post api), should exists, but > > > > doesn't]]] > > > > [2013-02-01 16:37:41,667][WARN][indices.cluster] [2] > > > > [TestDoc][2] failed to start shard > > > > org.elasticsearch.index.\*\*gateway. **IndexShardGatewayRecoveryExcep** tion: > > > > [TestDoc][2] shard allocated for local recovery (post api), should exists, > > > > but doesn't > > > > at org.elasticsearch.index.\*\*gateway.local. **LocalIndexShardGateway.** > > > > recover(\*\*LocalIndexShardGateway.java:\*\*108) > > > > at org.elasticsearch.index.\*\*gateway. **IndexShardGatewayService$1.** > > > > run(IndexShardGatewayService.\*\*java:177) > > > > at java.util.concurrent.\*\*ThreadPoolExecutor.runWorker(\*\*ThreadPoolExecutor.java:1110) > > > > > > at java.util.concurrent.\*\*ThreadPoolExecutor$Worker.run(\*\*ThreadPoolExecutor.java:603) > > > > > > at java.lang.Thread.run(Thread.**java:636) > > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > > > > sending failed shard for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > > s[INITIALIZING], reason [Failed to start shard, message [** > > > > IndexShardGatewayRecoveryExcep**tion[[TestDoc][2] shard allocated for > > > > local recovery (post api), should exists, but doesn't]]] > > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > > > > received shard failed for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > > s[INITIALIZING], reason [Failed to start shard, message [** > > > > IndexShardGatewayRecoveryExcep**tion[[TestDoc][2] shard allocated for > > > > local recovery (post api), should exists, but doesn't]]] > > > > [2013-02-01 16:39:00,306][WARN][discovery.zen] [2] master > > > > should not receive new cluster state from [[1][IlJPr1CBTmKxSgHyHJ7brg][** > > > > inet[/10.190.209.134:9300]]] > > > > > > > > Node1- > > > > > > > > [2013-02-01 10:08:03,861][DEBUG][action.**search.type] [1] > > > > failed to reduce search > > > > org.elasticsearch.action.search.ReduceSearchPhaseException: > > > > Failed to execute phase [fetch], [reduce] > > > > at org.elasticsearch.action.search.type. > > > > TransportSearchQueryThenFetchAction$AsyncAction.finishHim( > > > > TransportSearchQueryThenFetchA**ction.java:177) > > > > at org.elasticsearch.action. **search.type.** > > > > TransportSearchQueryThenFetchA**ction$AsyncAction$3.onResult(** > > > > TransportSearchQueryThenFetchA**ction.java:155) > > > > at org.elasticsearch.action.search.type. > > > > TransportSearchQueryThenFetchA**ction$AsyncAction$3.onResult(\*\* > > > > TransportSearchQueryThenFetchA**ction.java:1) > > > > at org.elasticsearch.search.action.SearchServiceTransportAction. > > > > sendExecuteFetch(SearchServiceTransportAction.java:345) > > > > at org.elasticsearch.action.search.type. > > > > TransportSearchQueryThenFetchAction$AsyncAction.executeFetch( > > > > TransportSearchQueryThenFetchAction.java:149) > > > > at org.elasticsearch.action.search.type. > > > > TransportSearchQueryThenFetchAction$AsyncAction$2.run(** > > > > TransportSearchQueryThenFetchA\*\*ction.java:136) > > > > at java.util.concurrent. **ThreadPoolExecutor$Worker.** > > > > runTask(ThreadPoolExecutor.\*\*java:886) > > > > at java.util.concurrent.\*\*ThreadPoolExecutor$Worker.run(\*\*ThreadPoolExecutor.java:908) > > > > > > at java.lang.Thread.run(Thread.**java:662) > > > > Caused by: java.lang.ClassCastException: org.elasticsearch.search.** > > > > facet.termsstats.longs.\*\*InternalTermsStatsLongFacet cannot be cast to > > > > org.elasticsearch.plugin.**multifssearch.InternalTermsStatsStringFacetM > > > > ulti > > > > at org.elasticsearch.plugin.multifssearch. > > > > InternalTermsStatsStringFacetMulti.reduce( > > > > InternalTermsStatsStringFacetM**ulti.java:490) > > > > at org.elasticsearch.plugin. **multifssearch.** > > > > TermsStatsFacetProcessorMulti.\*\*reduce(\*\*TermsStatsFacetProcessorMulti.\*\*java:166) > > > > > > at org.elasticsearch.search. **controller.** > > > > SearchPhaseController.merge(**SearchPhaseController.java:296) > > > > at org.elasticsearch.action.search.type. > > > > TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim( > > > > TransportSearchQueryThenFetchA**ction.java:190) > > > > at org.elasticsearch.action. **search.type.** > > > > TransportSearchQueryThenFetchA**ction$AsyncAction.finishHim(** > > > > TransportSearchQueryThenFetchA\*\*ction.java:175) > > > > ... 8 more > > > > [2013-02-01 12:59:42,516][INFO][cluster.metadata] [1] > > > > [TestDoc2] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > > [TestDoc2~type1] > > > > [2013-02-01 13:00:34,555][INFO][cluster.metadata] [1] > > > > [TestDoc3] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > > [TestDoc3~type1] > > > > > > > > The Cluster health api from node1; > > > > > > > > { > > > > "cluster\_name" : "test1", > > > > "status" : "red", > > > > "timed\_out" : false, > > > > "number\_of\_nodes" : 1, > > > > "number\_of\_data\_nodes" : 1, > > > > "active\_primary\_shards" : 4256, > > > > "active\_shards" : 4256, > > > > "relocating\_shards" : 0, > > > > "initializing\_shards" : 0, > > > > "unassigned\_shards" : 4209 > > > > } > > > > > > > > The Cluster health api from node2; > > > > { > > > > "cluster\_name" : "test", > > > > "status" : "red", > > > > "timed\_out" : false, > > > > "number\_of\_nodes" : 2, > > > > "number\_of\_data\_nodes" : 2, > > > > "active\_primary\_shards" : 8471, > > > > "active\_shards" : 8471, > > > > "relocating\_shards" : 0, > > > > "initializing\_shards" : 4, > > > > "unassigned\_shards" : 0 > > > > } > > > > > > > > I looked through the ES group but could not find the exact issue. > > > > It looks like one of the node ( primary) left the cluster because of > > > > the network issue( not sure what was the issue, assuming network issue). > > > > And the secondary got elected as master. And when the network issue was > > > > resolved. The primary node was trying to join the cluster, which did > > > > happen. But probably the state was not synched? or there two master nodes > > > > master1- having two node in cluster, but not able to communicate with data > > > > node. master2- having only one node in cluster. > > > > > > > > Please help me as this is going crazy over my head. I looked through > > > > the different threads, but nothing concrete. > > > > > > > > ## Thanks in advance Amit > > > > > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Amit](https://avatars.discourse-cdn.com/v4/letter/a/ee7513/32.png) [@Amit](https://discuss.elastic.co/u/Amit)\ **Post date:** [February 13, 2013, 8:56am UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594/7 "2013-02-13T08:56:03Z")
Hi Radu, Thanks for the response. As per the suggestion I have made the changes in the ES config "minimum\_master\_nodes" to \*2. \*And since it was time to add a new node in cluster. I added one more node in cluster ( total 3 nodes) and "minimum\_master\_nodes" to _2._ \* \* All my nodes have some configuration. Java max heap- _12g_ ; memory available on system -_16g (AWS - XLarge)_ Data drives to hold the data- _4 drives - 500gb each_. \* \* 1-When I restarted the cluster, lot of shards got relocated to the new node and when the cluster health became stable. The new node ( say node3 ) was holding data anywhere between 45-50 % of the total data. I assumed the data distribution among the nodes to be uniform? 2- When the system is creating new Indexes. The average load for the new node3 is going very high and is constantly between _100-200_. When I compare the size of the data across all the node, 85-90 % of the new index data goes to the new node3? 3- Change in the lucene merge setting will help or not. like index.merge.policy.max\_merge\_at\_once setting this to a higher value or index.merge.policy.min\_merge\_size to a higher value. Please help me understand the above. I read through the post; > **[Recent Code Search Outages](https://github.blog/2013-02-04-recent-code-search-outages/)** > > Last week, between Thursday, January 24 and Friday, January 25 we experienced a critical outage surrounding our newly-launched Code Search service. As always, we strive to provide detailed, transparent post-mortems about these incidents. We’ll do our... I am using ES 0.19.4 version and oracle java 1.6.0\_31. Is this the best combination or do I need to change the java version to 7 or any other version of java. I cannot change the es vesrion, as i have dependency around it. When I add a new node to a live cluster, what is expected behavior of ES. I mean since the es will be busy relocating the shards. What will the impact on indexing new data and performing search on the existing data. My apology for the lengthy post as I did not expected it to be! Thanks Amit On Tuesday, February 5, 2013 12:56:20 AM UTC+5:30, Radu Gheorghe wrote: > Hi, > > For two nodes you should have minimum\_master\_nodes=2, otherwise you can > end up with 2 1-node clusters. And 2/2+1=2 > On Feb 3, 2013 7:09 AM, "Amit Singh" \<[amitsi...@gmail.com](mailto:amitsi...@gmail.com) \\> > wrote: > > > Thanks Jorg and kimchy, > > > > I am using ES 0.19.4 version and minimum\_master\_nodes setting is default. > > Since I have only two nodes in the cluster, I did not change > > the minimum\_master\_nodes, as N/2+1 will give me value of 1 for two nodes. > > Further when I restarted the cluster I am still getting the same error! > > > > Please suggest. > > > > Thanks > > Amit > > > > On Sunday, February 3, 2013 3:42:02 AM UTC+5:30, kimchy wrote: > > > > > Jorg mentioned the important minimum\_master\_nodes, which version of ES > > > are you on? > > > > > > On Feb 1, 2013, at 11:18 PM, Jörg Prante [joerg...@gmail.com](mailto:joerg...@gmail.com) wrote: > > > > > > > Yes, it looks like a crash or a split. The problematic indexes are > > > > lost, and should be deleted. Otherwise, ES is not able to resolve the > > > > conflict. Note, there are precautions against such node splits, did you > > > > change the default settings in zen discovery, minimum\_master\_nodes for > > > > example? > > > > > > > > Best regards, > > > > > > > > Jörg > > > > > > > > Am 01.02.13 19:01, schrieb Amit Singh: > > > > > > > > > Hi All, > > > > > > > > > > I am having an ES cluster with 2 nodes. I am not as to what caused > > > > > this issue; > > > > > > > > > > Node 2- > > > > > > > > > > [2] received shard failed for [TestDocTestDoc][2], > > > > > node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start > > > > > shard, message [**IndexShardGatewayRecoveryExcep** tion[[TestDoc][2] > > > > > shard allocated for local recovery (post api), should exists, but > > > > > doesn't]]] > > > > > [2013-02-01 16:37:41,667][WARN][indices.cluster] [2] > > > > > [TestDoc][2] failed to start shard > > > > > org.elasticsearch.index.\*\*gateway. **IndexShardGatewayRecoveryExcep** tion: > > > > > [TestDoc][2] shard allocated for local recovery (post api), should exists, > > > > > but doesn't > > > > > at org.elasticsearch.index.\*\*gateway.local.\*_LocalIndexShardGateway._ > > > > > \*recover(\*\*LocalIndexShardGateway.java:\*\*108) > > > > > at org.elasticsearch.index.\*\*gateway. **IndexShardGatewayService$1.** > > > > > run(IndexShardGatewayService.\*\*java:177) > > > > > at java.util.concurrent.\*\*ThreadPoolExecutor.runWorker(\*\*ThreadPoolExecutor.java:1110) > > > > > > > > at java.util.concurrent.\*\*ThreadPoolExecutor$Worker.run(\*\*ThreadPoolExecutor.java:603) > > > > > > > > at java.lang.Thread.run(Thread.**java:636) > > > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > > > > > sending failed shard for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > > > s[INITIALIZING], reason [Failed to start shard, message [** > > > > > IndexShardGatewayRecoveryExcep**tion[[TestDoc][2] shard allocated for > > > > > local recovery (post api), should exists, but doesn't]]] > > > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > > > > > received shard failed for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > > > s[INITIALIZING], reason [Failed to start shard, message [** > > > > > IndexShardGatewayRecoveryExcep\*\*tion[[TestDoc][2] shard allocated for > > > > > local recovery (post api), should exists, but doesn't]]] > > > > > [2013-02-01 16:39:00,306][WARN][discovery.zen] [2] > > > > > master should not receive new cluster state from > > > > > [[1][IlJPr1CBTmKxSgHyHJ7brg][\*\*inet[/10.190.209.134:9300]]] > > > > > > > > > > Node1- > > > > > > > > > > [2013-02-01 10:08:03,861][DEBUG][action.**search.type] [1] > > > > > failed to reduce search > > > > > org.elasticsearch.action.search.ReduceSearchPhaseException: > > > > > Failed to execute phase [fetch], [reduce] > > > > > at org.elasticsearch.action.search.type. > > > > > TransportSearchQueryThenFetchAction$AsyncAction.finishHim( > > > > > TransportSearchQueryThenFetchA**ction.java:177) > > > > > at org.elasticsearch.action. **search.type.** > > > > > TransportSearchQueryThenFetchA**ction$AsyncAction$3.onResult(** > > > > > TransportSearchQueryThenFetchA**ction.java:155) > > > > > at org.elasticsearch.action.search.type. > > > > > TransportSearchQueryThenFetchA**ction$AsyncAction$3.onResult(\*\* > > > > > TransportSearchQueryThenFetchA_ction.java:1) > > > > > at org.elasticsearch.search.action.SearchServiceTransportAction. > > > > > \*sendExecuteFetch(**SearchServiceTransportAction.java:345) > > > > > at org.elasticsearch.action.search.type. > > > > > TransportSearchQueryThenFetchAction$AsyncAction.executeFetch( > > > > > TransportSearchQueryThenFetchA**ction.java:149) > > > > > at org.elasticsearch.action. **search.type.** > > > > > TransportSearchQueryThenFetchAction$AsyncAction$2.run( > > > > > TransportSearchQueryThenFetchA_\*ction.java:136) > > > > > at java.util.concurrent. **ThreadPoolExecutor$Worker.** > > > > > runTask(ThreadPoolExecutor.\*\*java:886) > > > > > at java.util.concurrent.\*\*ThreadPoolExecutor$Worker.run(\*\*ThreadPoolExecutor.java:908) > > > > > > > > at java.lang.Thread.run(Thread.**java:662) > > > > > Caused by: java.lang.ClassCastException: org.elasticsearch.search.** > > > > > facet.termsstats.longs.**InternalTermsStatsLongFacet cannot be cast to > > > > > org.elasticsearch.plugin.multifssearch. > > > > > InternalTermsStatsStringFacetM**ulti > > > > > at org.elasticsearch.plugin. **multifssearch.** > > > > > InternalTermsStatsStringFacetM**ulti.reduce(** > > > > > InternalTermsStatsStringFacetM_ulti.java:490) > > > > > at org.elasticsearch.plugin. **multifssearch.** > > > > > TermsStatsFacetProcessorMulti.reduce(TermsStatsFacetProcessorMulti. > > > > > \*java:166) > > > > > at org.elasticsearch.search. **controller.** > > > > > SearchPhaseController.merge(**SearchPhaseController.java:296) > > > > > at org.elasticsearch.action.search.type. > > > > > TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim( > > > > > TransportSearchQueryThenFetchA**ction.java:190) > > > > > at org.elasticsearch.action. **search.type.** > > > > > TransportSearchQueryThenFetchAction$AsyncAction.finishHim( > > > > > TransportSearchQueryThenFetchA_\*ction.java:175) > > > > > ... 8 more > > > > > [2013-02-01 12:59:42,516][INFO][cluster.metadata] [1] > > > > > [TestDoc2] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > > > [TestDoc2~type1] > > > > > [2013-02-01 13:00:34,555][INFO][cluster.metadata] [1] > > > > > [TestDoc3] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > > > [TestDoc3~type1] > > > > > > > > > > The Cluster health api from node1; > > > > > > > > > > { > > > > > "cluster\_name" : "test1", > > > > > "status" : "red", > > > > > "timed\_out" : false, > > > > > "number\_of\_nodes" : 1, > > > > > "number\_of\_data\_nodes" : 1, > > > > > "active\_primary\_shards" : 4256, > > > > > "active\_shards" : 4256, > > > > > "relocating\_shards" : 0, > > > > > "initializing\_shards" : 0, > > > > > "unassigned\_shards" : 4209 > > > > > } > > > > > > > > > > The Cluster health api from node2; > > > > > { > > > > > "cluster\_name" : "test", > > > > > "status" : "red", > > > > > "timed\_out" : false, > > > > > "number\_of\_nodes" : 2, > > > > > "number\_of\_data\_nodes" : 2, > > > > > "active\_primary\_shards" : 8471, > > > > > "active\_shards" : 8471, > > > > > "relocating\_shards" : 0, > > > > > "initializing\_shards" : 4, > > > > > "unassigned\_shards" : 0 > > > > > } > > > > > > > > > > I looked through the ES group but could not find the exact issue. > > > > > It looks like one of the node ( primary) left the cluster because of > > > > > the network issue( not sure what was the issue, assuming network issue). > > > > > And the secondary got elected as master. And when the network issue was > > > > > resolved. The primary node was trying to join the cluster, which did > > > > > happen. But probably the state was not synched? or there two master nodes > > > > > master1- having two node in cluster, but not able to communicate with data > > > > > node. master2- having only one node in cluster. > > > > > > > > > > Please help me as this is going crazy over my head. I looked through > > > > > the different threads, but nothing concrete. > > > > > > > > > > ## Thanks in advance Amit > > > > > > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > send an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [February 13, 2013, 2:12pm UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594/8 "2013-02-13T14:12:43Z")
Hello Amit, Normally, Elasticsearch tries to balance the number of shards across nodes. It doesn't look at how much data is in these shards or which index the shards belong to. That might explain your situation, but I'm not sure. If it doesn't make sense to you, please say some more about your index setup. Stuff like how many indices you have, how many shards per index, which kind of documents go in which index and what's the size of each shard. The good news is you can configure Elasticsearch to allocate shards in various ways. Take at these links: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... Although I think the last one is not available in 0.19.4. ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Wed, Feb 13, 2013 at 10:56 AM, Amit Singh [amitsingh.kec@gmail.com](mailto:amitsingh.kec@gmail.com)wrote: > Hi Radu, > > Thanks for the response. > > As per the suggestion I have made the changes in the ES config > "minimum\_master\_nodes" to \*2. \*And since it was time to add a new node in > cluster. I added one more node in cluster ( total 3 nodes) > and "minimum\_master\_nodes" to _2._ > \* > \* > All my nodes have some configuration. > Java max heap- _12g_ ; memory available on system -_16g (AWS - XLarge)_ > Data drives to hold the data- _4 drives - 500gb each_. > \* > \* > 1-When I restarted the cluster, lot of shards got relocated to the new > node and when the cluster health became stable. The new node ( say node3 ) > was holding data anywhere between 45-50 % of the total data. I assumed the > data distribution among the nodes to be uniform? > > 2- When the system is creating new Indexes. The average load for the new > node3 is going very high and is constantly between _100-200_. When I > compare the size of the data across all the node, 85-90 % of the new index > data goes to the new node3? > > 3- Change in the lucene merge setting will help or not. > like index.merge.policy.max\_merge\_at\_once setting this to a higher value or > index.merge.policy.min\_merge\_size to a higher value. > > Please help me understand the above. > > I read through the post; > [Recent Code Search Outages - The GitHub Blog](https://github.com/blog/1397-recent-code-search-outages) > > I am using ES 0.19.4 version and oracle java 1.6.0\_31. Is this the best > combination or do I need to change the java version to 7 or any other > version of java. I cannot change the es vesrion, as i have dependency > around it. > > When I add a new node to a live cluster, what is expected behavior of ES. > I mean since the es will be busy relocating the shards. What will the > impact on indexing new data and performing search on the existing data. > > My apology for the lengthy post as I did not expected it to be! > > Thanks > Amit > > On Tuesday, February 5, 2013 12:56:20 AM UTC+5:30, Radu Gheorghe wrote: > > > Hi, > > > > For two nodes you should have minimum\_master\_nodes=2, otherwise you can > > end up with 2 1-node clusters. And 2/2+1=2 > > On Feb 3, 2013 7:09 AM, "Amit Singh" [amitsi...@gmail.com](mailto:amitsi...@gmail.com) wrote: > > > > > Thanks Jorg and kimchy, > > > > > > I am using ES 0.19.4 version and minimum\_master\_nodes setting is default. > > > Since I have only two nodes in the cluster, I did not change > > > the minimum\_master\_nodes, as N/2+1 will give me value of 1 for two nodes. > > > Further when I restarted the cluster I am still getting the same error! > > > > > > Please suggest. > > > > > > Thanks > > > Amit > > > > > > On Sunday, February 3, 2013 3:42:02 AM UTC+5:30, kimchy wrote: > > > > > > > Jorg mentioned the important minimum\_master\_nodes, which version of ES > > > > are you on? > > > > > > > > On Feb 1, 2013, at 11:18 PM, Jörg Prante [joerg...@gmail.com](mailto:joerg...@gmail.com) wrote: > > > > > > > > > Yes, it looks like a crash or a split. The problematic indexes are > > > > > lost, and should be deleted. Otherwise, ES is not able to resolve the > > > > > conflict. Note, there are precautions against such node splits, did you > > > > > change the default settings in zen discovery, minimum\_master\_nodes for > > > > > example? > > > > > > > > > > Best regards, > > > > > > > > > > Jörg > > > > > > > > > > Am 01.02.13 19:01, schrieb Amit Singh: > > > > > > > > > > > Hi All, > > > > > > > > > > > > I am having an ES cluster with 2 nodes. I am not as to what caused > > > > > > this issue; > > > > > > > > > > > > Node 2- > > > > > > > > > > > > [2] received shard failed for [TestDocTestDoc][2], > > > > > > node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start > > > > > > shard, message [**IndexShardGatewayRecoveryExce** p**tion[[TestDoc][2] > > > > > > shard allocated for local recovery (post api), should exists, but > > > > > > doesn't]]] > > > > > > [2013-02-01 16:37:41,667][WARN][indices.cluster] [2] > > > > > > [TestDoc][2] failed to start shard > > > > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryExc > > > > > > ep**tion: [TestDoc][2] shard allocated for local recovery (post api), > > > > > > should exists, but doesn't > > > > > > at org.elasticsearch.index. **gatewa** y.local.\*\* > > > > > > LocalIndexShardGateway\*\*.\*\*recover( **LocalIndexShardGatewa** y.java:\*\*108) > > > > > > > > > > at org.elasticsearch.index. **gatewa** y.**IndexShardGatewayService$1. > > > > > > run(IndexShardGatewayService.java:177) > > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(** > > > > > > ThreadPool**Executor.java:1110) > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(** > > > > > > ThreadPoo**lExecutor.java:603) > > > > > > at java.lang.Thread.run(Thread.java:636) > > > > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > > > > > > sending failed shard for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > > > > s[INITIALIZING], reason [Failed to start shard, message [** > > > > > > IndexShardGatewayRecoveryExce **p** tion[[TestDoc][2] shard allocated > > > > > > for local recovery (post api), should exists, but doesn't]]] > > > > > > [2013-02-01 16:37:41,834][WARN][cluster.action.shard] [2] > > > > > > received shard failed for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P], > > > > > > s[INITIALIZING], reason [Failed to start shard, message [\*\* > > > > > > IndexShardGatewayRecoveryExce **p** tion[[TestDoc][2] shard allocated > > > > > > for local recovery (post api), should exists, but doesn't]]] > > > > > > [2013-02-01 16:39:00,306][WARN][discovery.zen] [2] > > > > > > master should not receive new cluster state from > > > > > > [[1][IlJPr1CBTmKxSgHyHJ7brg][**i** net[/10.190.209.134:9300]]] > > > > > > > > > > > > Node1- > > > > > > > > > > > > [2013-02-01 10:08:03,861][DEBUG][action. **se** arch.type] [1] > > > > > > failed to reduce search > > > > > > org.elasticsearch.action. **searc** h.**ReduceSearchPhaseException: > > > > > > Failed to execute phase [fetch], [reduce] > > > > > > at org.elasticsearch.action.search.type.** > > > > > > TransportSearchQueryThe **nFetchA** ction$AsyncAction.**finishHim(** > > > > > > TransportSearchQuery **ThenFetchA** ction.java:177) > > > > > > at org.elasticsearch.action. **searc** h.type.\*\* > > > > > > TransportSearchQueryThe **nFetchA** ction$AsyncAction$3.**onResult(** > > > > > > TransportSearchQueryT **henFetchA** ction.java:155) > > > > > > at org.elasticsearch.action. **searc** h.type.\*\* > > > > > > TransportSearchQueryThe **nFetchA** ction$AsyncAction$3.**onResult(** > > > > > > TransportSearchQueryT **henFetchA** ction.java:1) > > > > > > at org.elasticsearch.search. **actio** n.\*\* > > > > > > SearchServiceTransportAction\*\*.**sendExecuteFetch(SearchServic > > > > > > eTransportAction.java:345) > > > > > > at org.elasticsearch.action.search.type. > > > > > > TransportSearchQueryThe **nFetchA** ction$AsyncAction.executeFetch(** > > > > > > TransportSearchQueryTh **enFetchA** ction.java:149) > > > > > > at org.elasticsearch.action. **searc** h.type.\*\* > > > > > > TransportSearchQueryThe **nFetchA** ction$AsyncAction$2.**run(** > > > > > > TransportSearchQueryThenFe **tchA** ction.java:136) > > > > > > at java.util.concurrent. **ThreadPoo** lExecutor$Worker.**runTask(** > > > > > > ThreadPoolExecutor.**java:886) > > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(** > > > > > > ThreadPoo**lExecutor.java:908) > > > > > > at java.lang.Thread.run(Thread.java:662) > > > > > > Caused by: java.lang.ClassCastException: org.elasticsearch.search.** > > > > > > facet\*\*.termsstats.longs. **InternalTerm** sStatsLongFacet cannot be > > > > > > cast to org.elasticsearch.plugin. **multi** fssearch.\*\* > > > > > > InternalTermsStatsStr **ingFacetM** ulti > > > > > > at org.elasticsearch.plugin. **multi** fssearch.\*\* > > > > > > InternalTermsStatsStr **ingFacetM** ulti.reduce( **InternalT** > > > > > > ermsStatsStringFacetM**ulti.java:490) > > > > > > at org.elasticsearch.plugin.multifssearch. > > > > > > TermsStatsFacetProces**sorMulti._reduce(**TermsStatsFace** > > > > > > tProcessorMulti.java:166) > > > > > > at org.elasticsearch.search. **contr** oller.SearchPhaseController. > > > > > > \*merge(**SearchPhaseController.java:296) > > > > > > at org.elasticsearch.action.search.type. > > > > > > TransportSearchQueryThenFetchA **ction$AsyncAction.** inner**FinishHim( > > > > > > **TransportSearchQuery** ThenFetchAction.java:190) > > > > > > at org.elasticsearch.action. **searc** h.type. > > > > > > TransportSearchQueryThe __nFetchA__ ction$AsyncAction.**finishHim(** > > > > > > TransportSearchQuery__ThenFetchA_\*ction.java:175) > > > > > > ... 8 more > > > > > > [2013-02-01 12:59:42,516][INFO][cluster.metadata] [1] > > > > > > [TestDoc2] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > > > > [TestDoc2~type1] > > > > > > [2013-02-01 13:00:34,555][INFO][cluster.metadata] [1] > > > > > > [TestDoc3] creating index, cause [auto(bulk api)], shards [5]/[0], mappings > > > > > > [TestDoc3~type1] > > > > > > > > > > > > The Cluster health api from node1; > > > > > > > > > > > > { > > > > > > "cluster\_name" : "test1", > > > > > > "status" : "red", > > > > > > "timed\_out" : false, > > > > > > "number\_of\_nodes" : 1, > > > > > > "number\_of\_data\_nodes" : 1, > > > > > > "active\_primary\_shards" : 4256, > > > > > > "active\_shards" : 4256, > > > > > > "relocating\_shards" : 0, > > > > > > "initializing\_shards" : 0, > > > > > > "unassigned\_shards" : 4209 > > > > > > } > > > > > > > > > > > > The Cluster health api from node2; > > > > > > { > > > > > > "cluster\_name" : "test", > > > > > > "status" : "red", > > > > > > "timed\_out" : false, > > > > > > "number\_of\_nodes" : 2, > > > > > > "number\_of\_data\_nodes" : 2, > > > > > > "active\_primary\_shards" : 8471, > > > > > > "active\_shards" : 8471, > > > > > > "relocating\_shards" : 0, > > > > > > "initializing\_shards" : 4, > > > > > > "unassigned\_shards" : 0 > > > > > > } > > > > > > > > > > > > I looked through the ES group but could not find the exact issue. > > > > > > It looks like one of the node ( primary) left the cluster because of > > > > > > the network issue( not sure what was the issue, assuming network issue). > > > > > > And the secondary got elected as master. And when the network issue was > > > > > > resolved. The primary node was trying to join the cluster, which did > > > > > > happen. But probably the state was not synched? or there two master nodes > > > > > > master1- having two node in cluster, but not able to communicate with data > > > > > > node. master2- having only one node in cluster. > > > > > > > > > > > > Please help me as this is going crazy over my head. I looked through > > > > > > the different threads, but nothing concrete. > > > > > > > > > > > > ## Thanks in advance Amit > > > > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:51am UTC](https://discuss.elastic.co/t/very-weird-es-cluster-state-problem/10594/9 "2017-07-06T02:51:40Z")