InternalReplicaShardAllocator warnings after upgrade to 5.5.2

Dave_Konopka · September 12, 2017, 1:21pm

I recently upgraded a few clusters from version 5.2.2 -> 5.5.2. Since the upgrade I'm seeing a lot of InternalReplicaShardAllocator warnings in the Elasticsearch logs that seem to indicate a failing node. I'm not seeing nodes drop out of the cluster though or the cluster status go yellow/red. This seems to be happening in bursts every ~7-8 hours. Has anyone else run into this?

An example of the warning messages from the logs:

gist.github.com

https://gist.github.com/davekonopka/43ca9f9a7500911138c9f648a9d7ee02

elasticsearch-5-5-2-InternalReplicaShardAllocator.txt

[2017-09-11T07:18:15,361][WARN ][o.e.g.GatewayAllocator$InternalReplicaShardAllocator] [ip-10-xxx-xxx-xxx] [logs-cfn-init-2017.0
9.11][3]: failed to list shard for shard_store on node [bVMq_LXrRVyk_LHyFxUg8A]
org.elasticsearch.action.FailedNodeException: Failed node [bVMq_LXrRVyk_LHyFxUg8A]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.onFailure(TransportNodesAction.java:239) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction.access$200(TransportNodesAction.java:153) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.action.support.nodes.TransportNodesAction$AsyncAction$1.handleException(TransportNodesAction.java:211) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1067) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TcpTransport.lambda$handleException$16(TcpTransport.java:1467) ~[elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.common.util.concurrent.EsExecutors$1.execute(EsExecutors.java:110) [elasticsearch-5.5.2.jar:5.5.2]
        at org.elasticsearch.transport.TcpTransport.handleException(TcpTransport.java:1465) ~[elasticsearch-5.5.2.jar:5.5.2]

This file has been truncated. show original

dakrone · September 15, 2017, 4:41pm

From the log:

Caused by: java.io.FileNotFoundException: no segments* file found in store(mmapfs(/media/p_iops_vol0/elasticsearch/nodes/0/indices/eai7RWhVRzGa_FhlLJ73og/3/index)): files: [write.lock]

It looks like one of the shards is missing its data entirely, is it possible that the disk got corrupted or entirely filled and caused it not to be able to write the shard?

Regardless, for this particular shard, there is no data (since there are no segments in the directory), so you may need to force allocate an empty copy if you can't find a backup version to restore from.

system · October 13, 2017, 4:55pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.