InternalReplicaShardAllocator warnings after upgrade to 5.5.2

I recently upgraded a few clusters from version 5.2.2 -> 5.5.2. Since the upgrade I'm seeing a lot of InternalReplicaShardAllocator warnings in the Elasticsearch logs that seem to indicate a failing node. I'm not seeing nodes drop out of the cluster though or the cluster status go yellow/red. This seems to be happening in bursts every ~7-8 hours. Has anyone else run into this?

An example of the warning messages from the logs:

From the log:

Caused by: no segments* file found in store(mmapfs(/media/p_iops_vol0/elasticsearch/nodes/0/indices/eai7RWhVRzGa_FhlLJ73og/3/index)): files: [write.lock]

It looks like one of the shards is missing its data entirely, is it possible that the disk got corrupted or entirely filled and caused it not to be able to write the shard?

Regardless, for this particular shard, there is no data (since there are no segments in the directory), so you may need to force allocate an empty copy if you can't find a backup version to restore from.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.