Hi Folks,
Unluckily the disk on which ES was stored became full and one of the three nodes got shut down. Now the cluster is not able recover one of the shard with the following exception. Looks like ES is unable to read one of the ID of the document. Can someone help on how to restore the cluster:
[2015-09-16 15:38:36,686][WARN ][cluster.action.shard ] [es_prod_cortez_node2] [cortez][11] sending failed shard for [cortez][11], node[qnFTAqSRR4mMdPe96uUsXQ], [P], s[INITIALIZING], indexUUID [TIW3RodOR8-25fuy_1OuXw], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[cortez][11] failed to recover shard]; nested: ElasticsearchException[failed to read [data][xkYmcWMwleg1v2prraSSUA==]]; nested: ElasticsearchIllegalArgumentException[No version type match [108]]; ]]
[2015-09-16 15:38:38,388][WARN ][indices.cluster ] [es_prod_cortez_node2] [cortez][11] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [cortez][11] failed to recover shard
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:269)
at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchException: failed to read [data][xkYmcWMwleg1v2prraSSUA==]
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:511)
at org.elasticsearch.index.translog.TranslogStreams.readTranslogOperation(TranslogStreams.java:52)
at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:241)
... 4 more
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: No version type match [108]
at org.elasticsearch.index.VersionType.fromValue(VersionType.java:307)
at org.elasticsearch.index.translog.Translog$Index.readFrom(Translog.java:508)
... 6 more
Regards
Piyush