We're using Elasticsearch 1.2.2 in our cluster, and have 3 dedicated master
nodes. Last night, two of these master nodes (ES-SEARCH-64 and
ES-SEARCH-65) became primary master alternately in a period, and we found
quite a few 'marked shard as started, but shard has not been created, mark
shard as failed' in the logs, see an example below. Does anyone know what
this means? and why it happened?
[2014-09-10 06:58:15,567][DEBUG][action.admin.cluster.health]
[ES-SEARCH-64-master] connection exception while trying to forward request
to master node
[[ES-SEARCH-65-master][LF0n3IxmSFGcIG6FoqTqCw][es-search-65][inet[/10.0.4.5:9300]]{data=false,
AvailabilitySet=master, master=true}], scheduling a retry. Error:
[org.elasticsearch.transport.SendRequestTransportException:
[ES-SEARCH-65-master][inet[/10.0.4.5:9300]][cluster/health];
org.elasticsearch.transport.NodeNotConnectedException:
[ES-SEARCH-65-master][inet[/10.0.4.5:9300]] Node not connected]
[2014-09-10 06:58:16,130][DEBUG][action.admin.cluster.health]
[ES-SEARCH-64-master] connection exception while trying to forward request
to master node
[[ES-SEARCH-65-master][LF0n3IxmSFGcIG6FoqTqCw][es-search-65][inet[/10.0.4.5:9300]]{data=false,
AvailabilitySet=master, master=true}], scheduling a retry. Error:
[org.elasticsearch.transport.SendRequestTransportException:
[ES-SEARCH-65-master][inet[/10.0.4.5:9300]][cluster/health];
org.elasticsearch.transport.NodeNotConnectedException:
[ES-SEARCH-65-master][inet[/10.0.4.5:9300]] Node not connected]
[2014-09-10 06:58:17,045][DEBUG][action.admin.cluster.health]
[ES-SEARCH-64-master] observer: timeout notification from cluster service.
timeout setting [30s], time since start [30s]
[2014-09-10 06:58:18,342][DEBUG][action.admin.cluster.health]
[ES-SEARCH-64-master] connection exception while trying to forward request
to master node
[[ES-SEARCH-65-master][LF0n3IxmSFGcIG6FoqTqCw][es-search-65][inet[/10.0.4.5:9300]]{data=false,
AvailabilitySet=master, master=true}], scheduling a retry. Error:
[org.elasticsearch.transport.SendRequestTransportException:
[ES-SEARCH-65-master][inet[/10.0.4.5:9300]][cluster/health];
org.elasticsearch.transport.NodeNotConnectedException:
[ES-SEARCH-65-master][inet[/10.0.4.5:9300]] Node not connected]
[2014-09-10 06:58:18,734][DEBUG][action.admin.cluster.health]
[ES-SEARCH-64-master] connection exception while trying to forward request
to master node
[[ES-SEARCH-65-master][LF0n3IxmSFGcIG6FoqTqCw][es-search-65][inet[/10.0.4.5:9300]]{data=false,
AvailabilitySet=master, master=true}], scheduling a retry. Error:
[org.elasticsearch.transport.SendRequestTransportException:
[ES-SEARCH-65-master][inet[/10.0.4.5:9300]][cluster/health];
org.elasticsearch.transport.NodeNotConnectedException:
[ES-SEARCH-65-master][inet[/10.0.4.5:9300]] Node not connected]
[2014-09-10 06:58:20,093][WARN ][cluster.action.shard ]
[ES-SEARCH-64-master] [doc-v5][3] received shard failed for [doc-v5][3],
node[8-q2fmItSOyde6Yk2yUbWw], [R], s[STARTED], indexUUID
[fw1ocDrpSTejbY_m3WQP6Q], reason [master
[ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false,
AvailabilitySet=master, master=true} marked shard as started, but shard
has not been created, mark shard as failed]
[2014-09-10 06:58:20,093][WARN ][cluster.action.shard ]
[ES-SEARCH-64-master] [pub-v5][2] received shard failed for [pub-v5][2],
node[8-q2fmItSOyde6Yk2yUbWw], [P], s[STARTED], indexUUID
[4QlLH670Q_Kd6rb1WXhgaQ], reason [master
[ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false,
AvailabilitySet=master, master=true} marked shard as started, but shard
has not been created, mark shard as failed]
[2014-09-10 06:58:20,109][WARN ][cluster.action.shard ]
[ES-SEARCH-64-master] [doc-v5][23] received shard failed for [doc-v5][23],
node[_uM3zeD6QXaj3eWEzagADw], [R], s[STARTED], indexUUID
[fw1ocDrpSTejbY_m3WQP6Q], reason [master
[ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false,
AvailabilitySet=master, master=true} marked shard as started, but shard has
not been created, mark shard as failed]
[2014-09-10 06:58:20,109][WARN ][cluster.action.shard ]
[ES-SEARCH-64-master] [ppub-v5][16] received shard failed for
[ppub-v5][16], node[_uM3zeD6QXaj3eWEzagADw], [P], s[STARTED], indexUUID
[4QlLH670Q_Kd6rb1WXhgaQ], reason [master
[ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false,
AvailabilitySet=master, master=true} marked shard as started, but shard has
not been created, mark shard as failed]
[2014-09-10 06:58:20,155][WARN ][cluster.action.shard ]
[ES-SEARCH-64-master] [doc-v5][20] received shard failed for [doc-v5][20],
node[3fvBrmUcQmyhQ5XniKs7lg], [R], s[STARTED], indexUUID
[fw1ocDrpSTejbY_m3WQP6Q], reason [master
[ES-SEARCH-64-master][_zIl76rXQVGzTKVs4LbsZQ][es-search-64][inet[/10.0.4.4:9300]]{data=false,
AvailabilitySet=master, master=true} marked shard as started, but shard has
not been created, mark shard as failed]
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f42e8767-6c80-46a7-bdb1-6e9251579428%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.