I have ES 1.7.3 cluster ("number_of_replicas": "1"). I removed some node from cluster for server maintenance by 'service elasticsearch stop'. Cluster began to move from yellow to green status. But suddenly cluster status became red. I found one shard become unassigned (unassigned primary and unassigned replica). I restarted cluster, but it did not help.
In logs I found:
Caused by: org.apache.lucene.index.CorruptIndexException: [mediator-2016.06.30][3] Preexisting corrupted index [corrupted_O4zBHut3R2yR-A4yoktb5g] caused by: IOException[failed engine (reason: [corrupt file (source: [recovery phase 1 (file transfer)])])]
CorruptIndexException[checksum failed (hardware problem?) : expected=1tdr2zz actual=9ai2rl (resource=name [_7c1.fdt], length [3212112282], checksum [1tdr2zz], writtenBy [4.10.4])]
Then I run
java -cp lucene-core*.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex .... - fix
it say
Segments file=segments_43 numSegments=32 version=4.10.4 format= userData={translog_id=1467244801773, sync_id=AVY1Jy1-bT-99_0R3iJM}
1 of 32: name=_ekm docCount=3027895
version=4.10.4
codec=Lucene410
compound=false
numFiles=12
size (MB)=4,747.111
diagnostics = {os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_66, lucene.version=4.10.4, mergeMaxNumSegments=-1, os.arch=amd64, source=merge, mergeFactor=10, os.version=3.19.0-30-generic, timestamp=1467275414034}
no deletions
test: open reader.........OK
test: check integrity.....OK
test: check live docs.....OK
test: fields..............OK [107 fields]
test: field norms.........OK [1 fields]
test: terms, freq, prox...OK [36345118 terms; 1169162617 terms/docs pairs; 460647260 tokens]
test: stored fields.......OK [6055790 total field count; avg 2 fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
test: docvalues...........OK [1 docvalues fields; 0 BINARY; 1 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET]
32 of 32: name=_18ai docCount=1
version=4.10.4
codec=Lucene410
compound=true
numFiles=3
size (MB)=0.022
diagnostics = {os=Linux, java.vendor=Oracle Corporation, java.version=1.8.0_66, lucene.version=4.10.4, os.arch=amd64, source=flush, os.version=3.19.0-30-generic, timestamp=1467353739123}
no deletions
test: open reader.........OK
test: check integrity.....OK
test: check live docs.....OK
test: fields..............OK [83 fields]
test: field norms.........OK [1 fields]
test: terms, freq, prox...OK [358 terms; 358 terms/docs pairs; 132 tokens]
test: stored fields.......OK [2 total field count; avg 2 fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq vector fields per doc]
test: docvalues...........OK [1 docvalues fields; 0 BINARY; 1 NUMERIC; 0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET]
No problems were detected with this index.
I have not any ideas how to fix it.
Any Ideas welcome. May be there is a bug.