Hi All,
I am having an ES cluster with 2 nodes. I am not as to what caused this
issue;
Node 2-
[2] received shard failed for [TestDocTestDoc][2],
node[J368dRSdRxOUkTOEqIOsHg], [P], s[INITIALIZING], reason [Failed to start
shard, message [IndexShardGatewayRecoveryException[[TestDoc][2] shard
allocated for local recovery (post api), should exists, but doesn't]]]
[2013-02-01 16:37:41,667][WARN ][indices.cluster ] [2]
[TestDoc][2] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[TestDoc][2] shard allocated for local recovery (post api), should exists,
but doesn't
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:108)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:177)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
[2013-02-01 16:37:41,834][WARN ][cluster.action.shard ] [2] sending
failed shard for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local
recovery (post api), should exists, but doesn't]]]
[2013-02-01 16:37:41,834][WARN ][cluster.action.shard ] [2] received
shard failed for [TestDoc][2], node[J368dRSdRxOUkTOEqIOsHg], [P],
s[INITIALIZING], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[TestDoc][2] shard allocated for local
recovery (post api), should exists, but doesn't]]]
[2013-02-01 16:39:00,306][WARN ][discovery.zen ] [2] master
should not receive new cluster state from
[[1][IlJPr1CBTmKxSgHyHJ7brg][inet[/10.190.209.134:9300]]]
Node1-
[2013-02-01 10:08:03,861][DEBUG][action.search.type ] [1] failed to
reduce search
org.elasticsearch.action.search.ReduceSearchPhaseException: Failed to
execute phase [fetch], [reduce]
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:177)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:155)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$3.onResult(TransportSearchQueryThenFetchAction.java:1)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:345)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.executeFetch(TransportSearchQueryThenFetchAction.java:149)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction$2.run(TransportSearchQueryThenFetchAction.java:136)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.lang.ClassCastException:
org.elasticsearch.search.facet.termsstats.longs.InternalTermsStatsLongFacet
cannot be cast to
org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti
at
org.elasticsearch.plugin.multifssearch.InternalTermsStatsStringFacetMulti.reduce(InternalTermsStatsStringFacetMulti.java:490)
at
org.elasticsearch.plugin.multifssearch.TermsStatsFacetProcessorMulti.reduce(TermsStatsFacetProcessorMulti.java:166)
at
org.elasticsearch.search.controller.SearchPhaseController.merge(SearchPhaseController.java:296)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.innerFinishHim(TransportSearchQueryThenFetchAction.java:190)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.finishHim(TransportSearchQueryThenFetchAction.java:175)
... 8 more
[2013-02-01 12:59:42,516][INFO ][cluster.metadata ] [1] [TestDoc2]
creating index, cause [auto(bulk api)], shards [5]/[0], mappings
[TestDoc2~type1]
[2013-02-01 13:00:34,555][INFO ][cluster.metadata ] [1] [TestDoc3]
creating index, cause [auto(bulk api)], shards [5]/[0], mappings
[TestDoc3~type1]
The Cluster health api from node1;
{
"cluster_name" : "test1",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 4256,
"active_shards" : 4256,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 4209
}
The Cluster health api from node2;
{
"cluster_name" : "test",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
"active_primary_shards" : 8471,
"active_shards" : 8471,
"relocating_shards" : 0,
"initializing_shards" : 4,
"unassigned_shards" : 0
}
I looked through the ES group but could not find the exact issue.
It looks like one of the node ( primary) left the cluster because of the
network issue( not sure what was the issue, assuming network issue). And
the secondary got elected as master. And when the network issue was
resolved. The primary node was trying to join the cluster, which did
happen. But probably the state was not synched? or there two master nodes
master1- having two node in cluster, but not able to communicate with data
node. master2- having only one node in cluster.
Please help me as this is going crazy over my head. I looked through the
different threads, but nothing concrete.
Thanks in advance
Amit
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.