Hi,
I have situation where after performing a restart of ES process, the cluster now stays in RED and I cannot manually/forcefully re-route the INITIALIZING shard to fix cluster's state. Is there any way to fix this ?
From my understanding, I am assuming the index name "paas-docs-ucf2c-2015.02.25" has 5 primary shards and 1 replica.
"curl http://localhost:9200/_cat/shards?v"
paas-docs-ucf2c-2015.02.25 4 p INITIALIZING 10.241.236.54 ucf2b-devops-espublic2
paas-docs-ucf2c-2015.02.25 4 r UNASSIGNED
paas-docs-ucf2c-2015.02.25 0 r STARTED 9595924 3gb 10.241.238.234 ucf2b-devops-espublic3
paas-docs-ucf2c-2015.02.25 0 p STARTED 9595924 3gb 10.241.236.54 ucf2b-devops-espublic2
paas-docs-ucf2c-2015.02.25 3 r STARTED 9589378 2.9gb 10.241.238.234 ucf2b-devops-espublic3
paas-docs-ucf2c-2015.02.25 3 p STARTED 9589378 2.9gb 10.241.236.54 ucf2b-devops-espublic2
paas-docs-ucf2c-2015.02.25 1 r STARTED 9567744 2.9gb 10.241.238.234 ucf2b-devops-espublic3
paas-docs-ucf2c-2015.02.25 1 p STARTED 9567744 2.9gb 10.241.236.54 ucf2b-devops-espublic2
paas-docs-ucf2c-2015.02.25 2 r STARTED 9585661 3gb 10.241.238.234 ucf2b-devops-espublic3
paas-docs-ucf2c-2015.02.25 2 p STARTED 9585661 3gb 10.241.236.54 ucf2b-devops-espublic2
Summary:-
paas-docs-ucf2c-2015.02.25 Primary Shard [0] & its replica seems OK
paas-docs-ucf2c-2015.02.25 Primary Shard [1] & its replica seems OK
paas-docs-ucf2c-2015.02.25 Primary Shard [2] & its replica seems OK
paas-docs-ucf2c-2015.02.25 Primary Shard [3] & its replica seems OK
paas-docs-ucf2c-2015.02.25 Primary Shard [4] seems to be in "INITIALIZING" since many days and due to which its replica is in "UNASSIGNED" state.
Tailing logs shows below:-
[2015-03-04 08:20:05,588][WARN ][cluster.action.shard ] [ucf2b-devops-espublic2] [paas-docs-ucf2c-2015.02.25][4] sending failed shard for [paas-docs-ucf2c-2015.02.25][4], node[4aCqtG66SMSymamyBlFufg], [P], s[INITIALIZING], indexUUID [3iztrXA7SFmiW2L8x_EXDg], reason [Failed to start shard, message [IndexShardGatewayRecoveryException[[paas-docs-ucf2c-2015.02.25][4] failed to recover shard]; nested: TranslogCorruptedException[translog corruption while reading from stream]; nested: ElasticsearchIllegalArgumentException[No version type match [50]]; ]]
Can you please help me to fix this.
Best regards,
Abdul