# ES upgrade 0.20.6 to 1.4.2 -\> CorruptIndexException and FileNotFoundException
**URL:** https://discuss.elastic.co/t/es-upgrade-0-20-6-to-1-4-2-corruptindexexception-and-filenotfoundexception/21440
**Category:** Elasticsearch
**Created:** [December 31, 2014, 2:35pm UTC](https://discuss.elastic.co/t/es-upgrade-0-20-6-to-1-4-2-corruptindexexception-and-filenotfoundexception/21440 "2014-12-31T14:35:39Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Georgeta\_Boanea](https://discuss.elastic.co/u/Georgeta_Boanea)
#### Post date: [December 31, 2014, 2:35pm UTC](https://discuss.elastic.co/t/es-upgrade-0-20-6-to-1-4-2-corruptindexexception-and-filenotfoundexception/21440/1 "2014-12-31T14:35:39Z")
Hi All,
I have a 5 nodes cluster. I updated the cluster from 0.20.6 to 1.4.2.
When I start the cluster with shard allocation disabled, it starts and goes
into a yellow state, all good. When I enable shard allocation WARN messages
are generated:
INFO |||||| elasticsearch[node1][clusterService#updateTask][T#1]
org.elasticsearch.cluster.routing.allocation.decider [node1] updating
[cluster.routing.allocation.disable\_allocation] from [true] to [false]
[2014-12-31 13:46:26.310 GMT] WARN ||||||
elasticsearch[node1][[transport\_server\_worker.default]][T#4]{New I/O worker
#21} org.elasticsearch.cluster.action.shard [node1] [index1][2] received
shard failed for [index1][2], node[x6PqV8RMS8eA9GmBMZwjNQ], [P],
s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file
detected source: [recovery phase
1]][RecoverFilesRecoveryException[[index1][2] Failed to transfer [69] files
with total size of [6.5mb]]; nested: CorruptIndexException[checksum failed
(hardware problem?) : expected=17tw8li actual=1ig9y12
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)]; ]]
[2014-12-31 13:46:35.504 GMT] WARN ||||||
elasticsearch[node1][[transport\_server\_worker.default]][T#14]{New I/O
worker #31} org.elasticsearch.cluster.action.shard [node1] [index2][0]
received shard failed for [index2][0], node[GORnFBrmQLOAvK294MUHgA], [P],
s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file
detected source: [recovery phase
1]][RecoverFilesRecoveryException[[index2][0] Failed to transfer [163]
files with total size of [238.1mb]]; nested: CorruptIndexException[checksum
failed (hardware problem?) : expected=ptu7cd actual=1jw7kx9
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@38c14092)]; ]]
[2014-12-31 13:46:36.777 GMT] WARN ||||||
elasticsearch[node1][[transport\_server\_worker.default]][T#15]{New I/O
worker #32} org.elasticsearch.cluster.action.shard [node1] [index2][0]
received shard failed for [index2][0], node[GORnFBrmQLOAvK294MUHgA], [P],
s[STARTED], indexUUID [_na_], reason [master
[node1][8zFPkXuvQQWJvErc458tFA][[dw1949demum.int.demandware.com](http://dw1949demum.int.demandware.com)][inet[/127.0.0.1:48003]]{local=false,
power\_zone=default} marked shard as started, but shard has not been
created, mark shard as failed]
[2014-12-31 13:46:36.792 GMT] WARN ||||||
elasticsearch[node1][[transport\_server\_worker.default]][T#14]{New I/O
worker #31} org.elasticsearch.cluster.action.shard [node1] [index1][2]
received shard failed for [index1][2], node[2mIDLcOcQJO4i73QHb7d6Q], [P],
s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[index1][2] failed recovery]; nested:
EngineCreationFailureException[[index1][2] failed to open reader on
writer]; nested: FileNotFoundException[No such file [\_5aa.tis]]; ]]
[2014-12-31 13:46:47.261 GMT] WARN ||||||
elasticsearch[node1][[transport\_server\_worker.default]][T#6]{New I/O worker
#23} org.elasticsearch.cluster.action.shard [node1] [index1][2] received
shard failed for [index1][2], node[x6PqV8RMS8eA9GmBMZwjNQ], [P],
s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[index1][2] failed to fetch index
version after copying it over]; nested: CorruptIndexException[[index1][2]
Preexisting corrupted index [corrupted\_gExs5fftSwmCWWgUKN6Wbg] caused by:
CorruptIndexException[checksum failed (hardware problem?) :
expected=17tw8li actual=1ig9y12
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)]
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
problem?) : expected=17tw8li actual=1ig9y12
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)
at
org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
at org.elasticsearch.index.store.Store.verify(Store.java:365)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Suppressed: org.elasticsearch.transport.RemoteTransportException:
[node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file\_chunk]
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
(hardware problem?) : expected=55hiu actual=16i3yt2
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@108f1be6)
at
org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
at org.elasticsearch.index.store.Store.verify(Store.java:365)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Suppressed: org.elasticsearch.transport.RemoteTransportException:
[node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file\_chunk]
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
(hardware problem?) : expected=fszvyi actual=dxgex6
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@5217714c)
at
org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
at org.elasticsearch.index.store.Store.verify(Store.java:365)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Suppressed: org.elasticsearch.transport.RemoteTransportException:
[node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file\_chunk]
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
(hardware problem?) : expected=zfwkmg actual=a7sfag
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@355f052b)
at
org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
at org.elasticsearch.index.store.Store.verify(Store.java:365)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
.....
On two nodes I get a FileNotFoundException. Before I enable the shard
allocation the files are present on 2 nodes, after enabling shard
allocation the 'problem' files are just on one node.
On the nodes with missing files the following WARN message is generated:
[2014-12-31 13:46:48.068 GMT] WARN ||||||
elasticsearch[node2][generic][T#4] org.elasticsearch.indices.cluster
[node2] [index2][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[index2][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException:
[index2][0] failed to open reader on writer
at
org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:321)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
... 3 more
Caused by: java.io.FileNotFoundException: No such file [\_ovi.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
at
org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:487)
at
org.apache.lucene.codecs.lucene3x.TermInfosReader.(TermInfosReader.java:115)
at
org.apache.lucene.codecs.lucene3x.Lucene3xFields.newTermInfosReader(Lucene3xFields.java:142)
at
org.apache.lucene.codecs.lucene3x.Lucene3xFields.(Lucene3xFields.java:88)
at
org.apache.lucene.codecs.lucene3x.Lucene3xPostingsFormat.fieldsProducer(Lucene3xPostingsFormat.java:62)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:120)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:108)
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:144)
at
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:238)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:104)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:422)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112)
at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at
org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1527)
at
org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:309)
... 6 more
Is there a way to resolve the checksum problem? Any idea why the files are
deleted?
Any help would be greatly appreciated 🙂
Thank you
Georgeta
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/193543ed-5703-4686-b460-4bd8680194c2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/193543ed-5703-4686-b460-4bd8680194c2%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Georgeta\_Boanea](https://discuss.elastic.co/u/Georgeta_Boanea)
#### Post date: [January 2, 2015, 11:10am UTC](https://discuss.elastic.co/t/es-upgrade-0-20-6-to-1-4-2-corruptindexexception-and-filenotfoundexception/21440/2 "2015-01-02T11:10:35Z")
Some more info.....
I started the nodes with 'index.shard.check\_on\_startup: true', this is the
output:
Node5:
[2015-01-02 10:22:38.988 GMT] WARN ||||||
elasticsearch[node5][generic][T#4] org.elasticsearch.cluster.action.shard
[node5] [index1][2] sending failed shard for [index1][2],
node[\_0KcBcS0Q9qkprrD2jdzRw], [P], s[INITIALIZING], indexUUID [_na_],
reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[index1][2] failed recovery]; nested:
EngineCreationFailureException[[index1][2] failed to open reader on
writer]; nested: FileNotFoundException[No such file [\_5aa.tis]]; ]]
[2015-01-02 10:22:39.034 GMT] WARN ||||||
elasticsearch[node5][generic][T#4] org.elasticsearch.index.shard.service
[node5] [index1][2] check index [failure]
Segments file=segments\_l3 numSegments=7 versions=[3.6.2 .. 4.10.2] format=
userData={translog\_id=1395148001424}
1 of 7: name=\_5aa docCount=6094
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_5aa.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2 of 7: name=\_5ab docCount=2
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_5ab.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
3 of 7: name=\_5ad docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_5ad.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
4 of 7: name=\_5ag docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_5ag.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
5 of 7: name=\_5ah docCount=7
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_5ah.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
6 of 7: name=\_5aj docCount=2
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_5aj.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
7 of 7: name=\_5ak docCount=5
version=4.10.2
codec=Lucene410
compound=true
numFiles=4
size (MB)=0.012
diagnostics = {timestamp=1420193731527, os=Mac OS X,
os.version=10.10.1, source=flush, lucene.version=4.10.2, os.arch=x86\_64,
java.version=1.7.0\_71, java.vendor=Oracle Corporation}
has deletions [delGen=1]
test: open reader.........OK
test: check integrity.....OK
test: check live docs.....OK [4 deleted docs]
test: fields..............OK [49 fields]
test: field norms.........OK [14 fields]
test: terms, freq, prox...OK [81 terms; 81 terms/docs pairs; 20 tokens]
test (ignoring deletes): terms, freq, prox...OK [85 terms; 374
terms/docs pairs; 90 tokens]
test: stored fields.......OK [4 total field count; avg 4 fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
test: docvalues...........OK [1 docvalues fields; 0 BINARY; 1 NUMERIC;
0 SORTED; 0 SORTED\_NUMERIC; 0 SORTED\_SET]
WARNING: 6 broken segments (containing 6107 documents) detected
Node2:
[2015-01-02 10:23:13.186 GMT] WARN ||||||
elasticsearch[node2][generic][T#6] org.elasticsearch.index.shard.service
[node2] [index2][0] check index [failure]
Segments file=segments\_st numSegments=16 versions=[3.6.2 .. 4.10.2] format=
userData={translog\_id=1395148001180}
1 of 16: name=\_ovi docCount=116188
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_ovi.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2 of 16: name=\_lti docCount=110588
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_lti.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
3 of 16: name=\_l3c docCount=41007
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_l3c.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
4 of 16: name=\_q7i docCount=16411
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_q7i.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
5 of 16: name=\_twi docCount=9301
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_twi.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
6 of 16: name=\_rkj docCount=14930
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_rkj.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
7 of 16: name=\_ujc docCount=881
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_ujc.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
8 of 16: name=\_uip docCount=835
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_uip.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
9 of 16: name=\_un5 docCount=846
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_un5.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
10 of 16: name=\_uiz docCount=910
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_uiz.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
11 of 16: name=\_uni docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_uni.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
12 of 16: name=\_uno docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_uno.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
13 of 16: name=\_unp docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_unp.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14 of 16: name=\_unt docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_unt.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15 of 16: name=\_uo0 docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [\_uo0.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16 of 16: name=\_uo1 docCount=1
version=4.10.2
codec=Lucene410
compound=true
numFiles=3
size (MB)=0.009
diagnostics = {timestamp=1420193744623, os=Mac OS X,
os.version=10.10.1, source=flush, lucene.version=4.10.2, os.arch=x86\_64,
java.version=1.7.0\_71, java.vendor=Oracle Corporation}
no deletions
test: open reader.........OK
test: check integrity.....OK
test: check live docs.....OK
test: fields..............OK [37 fields]
test: field norms.........OK [12 fields]
test: terms, freq, prox...OK [69 terms; 69 terms/docs pairs; 35 tokens]
test: stored fields.......OK [4 total field count; avg 4 fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
test: docvalues...........OK [1 docvalues fields; 0 BINARY; 1 NUMERIC;
0 SORTED; 0 SORTED\_NUMERIC; 0 SORTED\_SET]
WARNING: 15 broken segments (containing 311902 documents) detected
On Wednesday, December 31, 2014 3:35:39 PM UTC+1, Georgeta wrote:
> Hi All,
>
> I have a 5 nodes cluster. I updated the cluster from 0.20.6 to 1.4.2.
> When I start the cluster with shard allocation disabled, it starts and
> goes into a yellow state, all good. When I enable shard allocation WARN
> messages are generated:
>
> INFO |||||| elasticsearch[node1][clusterService#updateTask][T#1]
> org.elasticsearch.cluster.routing.allocation.decider [node1] updating
> [cluster.routing.allocation.disable\_allocation] from [true] to [false]
>
> [2014-12-31 13:46:26.310 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#4]{New I/O worker
> #21} org.elasticsearch.cluster.action.shard [node1] [index1][2] received
> shard failed for [index1][2], node[x6PqV8RMS8eA9GmBMZwjNQ], [P],
> s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file
> detected source: [recovery phase
> 1]][RecoverFilesRecoveryException[[index1][2] Failed to transfer [69] files
> with total size of [6.5mb]]; nested: CorruptIndexException[checksum failed
> (hardware problem?) : expected=17tw8li actual=1ig9y12
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)]; ]]
>
> [2014-12-31 13:46:35.504 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#14]{New I/O
> worker #31} org.elasticsearch.cluster.action.shard [node1] [index2][0]
> received shard failed for [index2][0], node[GORnFBrmQLOAvK294MUHgA], [P],
> s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file
> detected source: [recovery phase
> 1]][RecoverFilesRecoveryException[[index2][0] Failed to transfer [163]
> files with total size of [238.1mb]]; nested: CorruptIndexException[checksum
> failed (hardware problem?) : expected=ptu7cd actual=1jw7kx9
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@38c14092)]; ]]
>
> [2014-12-31 13:46:36.777 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#15]{New I/O
> worker #32} org.elasticsearch.cluster.action.shard [node1] [index2][0]
> received shard failed for [index2][0], node[GORnFBrmQLOAvK294MUHgA], [P],
> s[STARTED], indexUUID [_na_], reason [master
> [node1][8zFPkXuvQQWJvErc458tFA][[dw1949demum.int.demandware.com](http://dw1949demum.int.demandware.com)][inet[/127.0.0.1:48003]]{local=false,
> power\_zone=default} marked shard as started, but shard has not been
> created, mark shard as failed]
>
> [2014-12-31 13:46:36.792 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#14]{New I/O
> worker #31} org.elasticsearch.cluster.action.shard [node1] [index1][2]
> received shard failed for [index1][2], node[2mIDLcOcQJO4i73QHb7d6Q], [P],
> s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message
> [IndexShardGatewayRecoveryException[[index1][2] failed recovery]; nested:
> EngineCreationFailureException[[index1][2] failed to open reader on
> writer]; nested: FileNotFoundException[No such file [\_5aa.tis]]; ]]
>
> [2014-12-31 13:46:47.261 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#6]{New I/O worker
> #23} org.elasticsearch.cluster.action.shard [node1] [index1][2] received
> shard failed for [index1][2], node[x6PqV8RMS8eA9GmBMZwjNQ], [P],
> s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message
> [IndexShardGatewayRecoveryException[[index1][2] failed to fetch index
> version after copying it over]; nested: CorruptIndexException[[index1][2]
> Preexisting corrupted index [corrupted\_gExs5fftSwmCWWgUKN6Wbg] caused by:
> CorruptIndexException[checksum failed (hardware problem?) :
> expected=17tw8li actual=1ig9y12
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)]
> org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
> problem?) : expected=17tw8li actual=1ig9y12
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)
> at
> org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
> at org.elasticsearch.index.store.Store.verify(Store.java:365)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Suppressed: org.elasticsearch.transport.RemoteTransportException:
> [node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file\_chunk]
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> (hardware problem?) : expected=55hiu actual=16i3yt2
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@108f1be6)
> at
> org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
> at org.elasticsearch.index.store.Store.verify(Store.java:365)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Suppressed: org.elasticsearch.transport.RemoteTransportException:
> [node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file\_chunk]
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> (hardware problem?) : expected=fszvyi actual=dxgex6
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@5217714c)
> at
> org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
> at org.elasticsearch.index.store.Store.verify(Store.java:365)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Suppressed: org.elasticsearch.transport.RemoteTransportException:
> [node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file\_chunk]
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> (hardware problem?) : expected=zfwkmg actual=a7sfag
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@355f052b)
> at
> org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
> at org.elasticsearch.index.store.Store.verify(Store.java:365)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> .....
>
> On two nodes I get a FileNotFoundException. Before I enable the shard
> allocation the files are present on 2 nodes, after enabling shard
> allocation the 'problem' files are just on one node.
> On the nodes with missing files the following WARN message is generated:
>
> [2014-12-31 13:46:48.068 GMT] WARN ||||||
> elasticsearch[node2][generic][T#4] org.elasticsearch.indices.cluster
> [node2] [index2][0] failed to start shard
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
> [index2][0] failed recovery
> at
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.elasticsearch.index.engine.EngineCreationFailureException:
> [index2][0] failed to open reader on writer
> at
> org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:321)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
> at
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
> at
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
> ... 3 more
> Caused by: java.io.FileNotFoundException: No such file [\_ovi.tis]
> at
> org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
> at
> org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
> at
> org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
> at
> org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
> at
> org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:487)
> at
> org.apache.lucene.codecs.lucene3x.TermInfosReader.(TermInfosReader.java:115)
> at
> org.apache.lucene.codecs.lucene3x.Lucene3xFields.newTermInfosReader(Lucene3xFields.java:142)
> at
> org.apache.lucene.codecs.lucene3x.Lucene3xFields.(Lucene3xFields.java:88)
> at
> org.apache.lucene.codecs.lucene3x.Lucene3xPostingsFormat.fieldsProducer(Lucene3xPostingsFormat.java:62)
> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:120)
> at org.apache.lucene.index.SegmentReader.(SegmentReader.java:108)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:144)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:238)
> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:104)
> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:422)
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112)
> at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
> at
> org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1527)
> at
> org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:309)
> ... 6 more
>
> Is there a way to resolve the checksum problem? Any idea why the files are
> deleted?
> Any help would be greatly appreciated 🙂
>
> Thank you
> Georgeta
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/1112b66f-c3bc-42fa-bd63-5757b1a6ebb6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/1112b66f-c3bc-42fa-bd63-5757b1a6ebb6%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@Georgeta\_Boanea](https://discuss.elastic.co/u/Georgeta_Boanea)
#### Post date: [January 5, 2015, 9:26am UTC](https://discuss.elastic.co/t/es-upgrade-0-20-6-to-1-4-2-corruptindexexception-and-filenotfoundexception/21440/3 "2015-01-05T09:26:34Z")
Any ideas?
On Wednesday, December 31, 2014 3:35:39 PM UTC+1, Georgeta wrote:
> Hi All,
>
> I have a 5 nodes cluster. I updated the cluster from 0.20.6 to 1.4.2.
> When I start the cluster with shard allocation disabled, it starts and
> goes into a yellow state, all good. When I enable shard allocation WARN
> messages are generated:
>
> INFO |||||| elasticsearch[node1][clusterService#updateTask][T#1]
> org.elasticsearch.cluster.routing.allocation.decider [node1] updating
> [cluster.routing.allocation.disable\_allocation] from [true] to [false]
>
> [2014-12-31 13:46:26.310 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#4]{New I/O worker
> #21} org.elasticsearch.cluster.action.shard [node1] [index1][2] received
> shard failed for [index1][2], node[x6PqV8RMS8eA9GmBMZwjNQ], [P],
> s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file
> detected source: [recovery phase
> 1]][RecoverFilesRecoveryException[[index1][2] Failed to transfer [69] files
> with total size of [6.5mb]]; nested: CorruptIndexException[checksum failed
> (hardware problem?) : expected=17tw8li actual=1ig9y12
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)]; ]]
>
> [2014-12-31 13:46:35.504 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#14]{New I/O
> worker #31} org.elasticsearch.cluster.action.shard [node1] [index2][0]
> received shard failed for [index2][0], node[GORnFBrmQLOAvK294MUHgA], [P],
> s[STARTED], indexUUID [_na_], reason [engine failure, message [corrupt file
> detected source: [recovery phase
> 1]][RecoverFilesRecoveryException[[index2][0] Failed to transfer [163]
> files with total size of [238.1mb]]; nested: CorruptIndexException[checksum
> failed (hardware problem?) : expected=ptu7cd actual=1jw7kx9
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@38c14092)]; ]]
>
> [2014-12-31 13:46:36.777 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#15]{New I/O
> worker #32} org.elasticsearch.cluster.action.shard [node1] [index2][0]
> received shard failed for [index2][0], node[GORnFBrmQLOAvK294MUHgA], [P],
> s[STARTED], indexUUID [_na_], reason [master
> [node1][8zFPkXuvQQWJvErc458tFA][[dw1949demum.int.demandware.com](http://dw1949demum.int.demandware.com)][inet[/127.0.0.1:48003]]{local=false,
> power\_zone=default} marked shard as started, but shard has not been
> created, mark shard as failed]
>
> [2014-12-31 13:46:36.792 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#14]{New I/O
> worker #31} org.elasticsearch.cluster.action.shard [node1] [index1][2]
> received shard failed for [index1][2], node[2mIDLcOcQJO4i73QHb7d6Q], [P],
> s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message
> [IndexShardGatewayRecoveryException[[index1][2] failed recovery]; nested:
> EngineCreationFailureException[[index1][2] failed to open reader on
> writer]; nested: FileNotFoundException[No such file [\_5aa.tis]]; ]]
>
> [2014-12-31 13:46:47.261 GMT] WARN ||||||
> elasticsearch[node1][[transport\_server\_worker.default]][T#6]{New I/O worker
> #23} org.elasticsearch.cluster.action.shard [node1] [index1][2] received
> shard failed for [index1][2], node[x6PqV8RMS8eA9GmBMZwjNQ], [P],
> s[INITIALIZING], indexUUID [_na_], reason [Failed to start shard, message
> [IndexShardGatewayRecoveryException[[index1][2] failed to fetch index
> version after copying it over]; nested: CorruptIndexException[[index1][2]
> Preexisting corrupted index [corrupted\_gExs5fftSwmCWWgUKN6Wbg] caused by:
> CorruptIndexException[checksum failed (hardware problem?) :
> expected=17tw8li actual=1ig9y12
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)]
> org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
> problem?) : expected=17tw8li actual=1ig9y12
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)
> at
> org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
> at org.elasticsearch.index.store.Store.verify(Store.java:365)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Suppressed: org.elasticsearch.transport.RemoteTransportException:
> [node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file\_chunk]
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> (hardware problem?) : expected=55hiu actual=16i3yt2
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@108f1be6)
> at
> org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
> at org.elasticsearch.index.store.Store.verify(Store.java:365)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Suppressed: org.elasticsearch.transport.RemoteTransportException:
> [node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file\_chunk]
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> (hardware problem?) : expected=fszvyi actual=dxgex6
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@5217714c)
> at
> org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
> at org.elasticsearch.index.store.Store.verify(Store.java:365)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Suppressed: org.elasticsearch.transport.RemoteTransportException:
> [node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file\_chunk]
> Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
> (hardware problem?) : expected=zfwkmg actual=a7sfag
> resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@355f052b)
> at
> org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
> at org.elasticsearch.index.store.Store.verify(Store.java:365)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
> at
> org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
> at
> org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> .....
>
> On two nodes I get a FileNotFoundException. Before I enable the shard
> allocation the files are present on 2 nodes, after enabling shard
> allocation the 'problem' files are just on one node.
> On the nodes with missing files the following WARN message is generated:
>
> [2014-12-31 13:46:48.068 GMT] WARN ||||||
> elasticsearch[node2][generic][T#4] org.elasticsearch.indices.cluster
> [node2] [index2][0] failed to start shard
> org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
> [index2][0] failed recovery
> at
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.elasticsearch.index.engine.EngineCreationFailureException:
> [index2][0] failed to open reader on writer
> at
> org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:321)
> at
> org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
> at
> org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
> at
> org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
> ... 3 more
> Caused by: java.io.FileNotFoundException: No such file [\_ovi.tis]
> at
> org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
> at
> org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
> at
> org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
> at
> org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
> at
> org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:487)
> at
> org.apache.lucene.codecs.lucene3x.TermInfosReader.(TermInfosReader.java:115)
> at
> org.apache.lucene.codecs.lucene3x.Lucene3xFields.newTermInfosReader(Lucene3xFields.java:142)
> at
> org.apache.lucene.codecs.lucene3x.Lucene3xFields.(Lucene3xFields.java:88)
> at
> org.apache.lucene.codecs.lucene3x.Lucene3xPostingsFormat.fieldsProducer(Lucene3xPostingsFormat.java:62)
> at
> org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:120)
> at org.apache.lucene.index.SegmentReader.(SegmentReader.java:108)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:144)
> at
> org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:238)
> at
> org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:104)
> at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:422)
> at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112)
> at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
> at
> org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1527)
> at
> org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:309)
> ... 6 more
>
> Is there a way to resolve the checksum problem? Any idea why the files are
> deleted?
> Any help would be greatly appreciated 🙂
>
> Thank you
> Georgeta
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d9c47639-00ce-4bd2-80da-cd105ab7e8d7%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d9c47639-00ce-4bd2-80da-cd105ab7e8d7%40googlegroups.com).
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 12:40am UTC](https://discuss.elastic.co/t/es-upgrade-0-20-6-to-1-4-2-corruptindexexception-and-filenotfoundexception/21440/4 "2017-07-06T00:40:57Z")