Some more info.....
I started the nodes with 'index.shard.check_on_startup: true', this is the
output:
Node5:
[2015-01-02 10:22:38.988 GMT] WARN ||||||
elasticsearch[node5][generic][T#4] org.elasticsearch.cluster.action.shard
[node5] [index1][2] sending failed shard for [index1][2],
node[_0KcBcS0Q9qkprrD2jdzRw], [P], s[INITIALIZING], indexUUID [na],
reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[index1][2] failed recovery]; nested:
EngineCreationFailureException[[index1][2] failed to open reader on
writer]; nested: FileNotFoundException[No such file [_5aa.tis]]; ]]
[2015-01-02 10:22:39.034 GMT] WARN ||||||
elasticsearch[node5][generic][T#4] org.elasticsearch.index.shard.service
[node5] [index1][2] check index [failure]
Segments file=segments_l3 numSegments=7 versions=[3.6.2 .. 4.10.2] format=
userData={translog_id=1395148001424}
1 of 7: name=_5aa docCount=6094
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_5aa.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2 of 7: name=_5ab docCount=2
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_5ab.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
3 of 7: name=_5ad docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_5ad.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
4 of 7: name=_5ag docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_5ag.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
5 of 7: name=_5ah docCount=7
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_5ah.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
6 of 7: name=_5aj docCount=2
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_5aj.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
7 of 7: name=_5ak docCount=5
version=4.10.2
codec=Lucene410
compound=true
numFiles=4
size (MB)=0.012
diagnostics = {timestamp=1420193731527, os=Mac OS X,
os.version=10.10.1, source=flush, lucene.version=4.10.2, os.arch=x86_64,
java.version=1.7.0_71, java.vendor=Oracle Corporation}
has deletions [delGen=1]
test: open reader.........OK
test: check integrity.....OK
test: check live docs.....OK [4 deleted docs]
test: fields..............OK [49 fields]
test: field norms.........OK [14 fields]
test: terms, freq, prox...OK [81 terms; 81 terms/docs pairs; 20 tokens]
test (ignoring deletes): terms, freq, prox...OK [85 terms; 374
terms/docs pairs; 90 tokens]
test: stored fields.......OK [4 total field count; avg 4 fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
test: docvalues...........OK [1 docvalues fields; 0 BINARY; 1 NUMERIC;
0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET]
WARNING: 6 broken segments (containing 6107 documents) detected
Node2:
[2015-01-02 10:23:13.186 GMT] WARN ||||||
elasticsearch[node2][generic][T#6] org.elasticsearch.index.shard.service
[node2] [index2][0] check index [failure]
Segments file=segments_st numSegments=16 versions=[3.6.2 .. 4.10.2] format=
userData={translog_id=1395148001180}
1 of 16: name=_ovi docCount=116188
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_ovi.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2 of 16: name=_lti docCount=110588
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_lti.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
3 of 16: name=_l3c docCount=41007
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_l3c.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
4 of 16: name=_q7i docCount=16411
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_q7i.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
5 of 16: name=_twi docCount=9301
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_twi.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
6 of 16: name=_rkj docCount=14930
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_rkj.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
7 of 16: name=_ujc docCount=881
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_ujc.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
8 of 16: name=_uip docCount=835
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_uip.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
9 of 16: name=_un5 docCount=846
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_un5.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
10 of 16: name=_uiz docCount=910
version=3.6.2
codec=Lucene3x
compound=false
numFiles=11
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_uiz.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
11 of 16: name=_uni docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_uni.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
12 of 16: name=_uno docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_uno.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
13 of 16: name=_unp docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_unp.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
14 of 16: name=_unt docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_unt.tii]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
15 of 16: name=_uo0 docCount=1
version=3.6.2
codec=Lucene3x
compound=false
numFiles=10
FAILED
WARNING: fixIndex() would remove reference to this segment; full
exception:
java.io.FileNotFoundException: No such file [_uo0.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.fileLength(DistributorDirectory.java:113)
at
org.apache.lucene.store.FilterDirectory.fileLength(FilterDirectory.java:63)
at
org.apache.lucene.index.SegmentCommitInfo.sizeInBytes(SegmentCommitInfo.java:219)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:561)
at org.apache.lucene.index.CheckIndex.checkIndex(CheckIndex.java:393)
at
org.elasticsearch.index.shard.service.InternalIndexShard.checkIndex(InternalIndexShard.java:1056)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:727)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
16 of 16: name=_uo1 docCount=1
version=4.10.2
codec=Lucene410
compound=true
numFiles=3
size (MB)=0.009
diagnostics = {timestamp=1420193744623, os=Mac OS X,
os.version=10.10.1, source=flush, lucene.version=4.10.2, os.arch=x86_64,
java.version=1.7.0_71, java.vendor=Oracle Corporation}
no deletions
test: open reader.........OK
test: check integrity.....OK
test: check live docs.....OK
test: fields..............OK [37 fields]
test: field norms.........OK [12 fields]
test: terms, freq, prox...OK [69 terms; 69 terms/docs pairs; 35 tokens]
test: stored fields.......OK [4 total field count; avg 4 fields per doc]
test: term vectors........OK [0 total vector count; avg 0 term/freq
vector fields per doc]
test: docvalues...........OK [1 docvalues fields; 0 BINARY; 1 NUMERIC;
0 SORTED; 0 SORTED_NUMERIC; 0 SORTED_SET]
WARNING: 15 broken segments (containing 311902 documents) detected
On Wednesday, December 31, 2014 3:35:39 PM UTC+1, Georgeta wrote:
Hi All,
I have a 5 nodes cluster. I updated the cluster from 0.20.6 to 1.4.2.
When I start the cluster with shard allocation disabled, it starts and
goes into a yellow state, all good. When I enable shard allocation WARN
messages are generated:
INFO |||||| elasticsearch[node1][clusterService#updateTask][T#1]
org.elasticsearch.cluster.routing.allocation.decider [node1] updating
[cluster.routing.allocation.disable_allocation] from [true] to [false]
[2014-12-31 13:46:26.310 GMT] WARN ||||||
elasticsearch[node1][[transport_server_worker.default]][T#4]{New I/O worker
#21} org.elasticsearch.cluster.action.shard [node1] [index1][2] received
shard failed for [index1][2], node[x6PqV8RMS8eA9GmBMZwjNQ], [P],
s[STARTED], indexUUID [na], reason [engine failure, message [corrupt file
detected source: [recovery phase
1]][RecoverFilesRecoveryException[[index1][2] Failed to transfer [69] files
with total size of [6.5mb]]; nested: CorruptIndexException[checksum failed
(hardware problem?) : expected=17tw8li actual=1ig9y12
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)]; ]]
[2014-12-31 13:46:35.504 GMT] WARN ||||||
elasticsearch[node1][[transport_server_worker.default]][T#14]{New I/O
worker #31} org.elasticsearch.cluster.action.shard [node1] [index2][0]
received shard failed for [index2][0], node[GORnFBrmQLOAvK294MUHgA], [P],
s[STARTED], indexUUID [na], reason [engine failure, message [corrupt file
detected source: [recovery phase
1]][RecoverFilesRecoveryException[[index2][0] Failed to transfer [163]
files with total size of [238.1mb]]; nested: CorruptIndexException[checksum
failed (hardware problem?) : expected=ptu7cd actual=1jw7kx9
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@38c14092)]; ]]
[2014-12-31 13:46:36.777 GMT] WARN ||||||
elasticsearch[node1][[transport_server_worker.default]][T#15]{New I/O
worker #32} org.elasticsearch.cluster.action.shard [node1] [index2][0]
received shard failed for [index2][0], node[GORnFBrmQLOAvK294MUHgA], [P],
s[STARTED], indexUUID [na], reason [master
[node1][8zFPkXuvQQWJvErc458tFA][dw1949demum.int.demandware.com][inet[/127.0.0.1:48003]]{local=false,
power_zone=default} marked shard as started, but shard has not been
created, mark shard as failed]
[2014-12-31 13:46:36.792 GMT] WARN ||||||
elasticsearch[node1][[transport_server_worker.default]][T#14]{New I/O
worker #31} org.elasticsearch.cluster.action.shard [node1] [index1][2]
received shard failed for [index1][2], node[2mIDLcOcQJO4i73QHb7d6Q], [P],
s[INITIALIZING], indexUUID [na], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[index1][2] failed recovery]; nested:
EngineCreationFailureException[[index1][2] failed to open reader on
writer]; nested: FileNotFoundException[No such file [_5aa.tis]]; ]]
[2014-12-31 13:46:47.261 GMT] WARN ||||||
elasticsearch[node1][[transport_server_worker.default]][T#6]{New I/O worker
#23} org.elasticsearch.cluster.action.shard [node1] [index1][2] received
shard failed for [index1][2], node[x6PqV8RMS8eA9GmBMZwjNQ], [P],
s[INITIALIZING], indexUUID [na], reason [Failed to start shard, message
[IndexShardGatewayRecoveryException[[index1][2] failed to fetch index
version after copying it over]; nested: CorruptIndexException[[index1][2]
Preexisting corrupted index [corrupted_gExs5fftSwmCWWgUKN6Wbg] caused by:
CorruptIndexException[checksum failed (hardware problem?) :
expected=17tw8li actual=1ig9y12
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)]
org.apache.lucene.index.CorruptIndexException: checksum failed (hardware
problem?) : expected=17tw8li actual=1ig9y12
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@61297ce5)
at
org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
at org.elasticsearch.index.store.Store.verify(Store.java:365)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Suppressed: org.elasticsearch.transport.RemoteTransportException:
[node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file_chunk]
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
(hardware problem?) : expected=55hiu actual=16i3yt2
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@108f1be6)
at
org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
at org.elasticsearch.index.store.Store.verify(Store.java:365)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Suppressed: org.elasticsearch.transport.RemoteTransportException:
[node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file_chunk]
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
(hardware problem?) : expected=fszvyi actual=dxgex6
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@5217714c)
at
org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
at org.elasticsearch.index.store.Store.verify(Store.java:365)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Suppressed: org.elasticsearch.transport.RemoteTransportException:
[node5][inet[/127.0.0.1:48043]][internal:index/shard/recovery/file_chunk]
Caused by: org.apache.lucene.index.CorruptIndexException: checksum failed
(hardware problem?) : expected=zfwkmg actual=a7sfag
resource=(org.apache.lucene.store.FSDirectory$FSIndexOutput@355f052b)
at
org.elasticsearch.index.store.LegacyVerification$Adler32VerifyingIndexOutput.verify(LegacyVerification.java:73)
at org.elasticsearch.index.store.Store.verify(Store.java:365)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:599)
at
org.elasticsearch.indices.recovery.RecoveryTarget$FileChunkTransportRequestHandler.messageReceived(RecoveryTarget.java:536)
at
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:275)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
.....
On two nodes I get a FileNotFoundException. Before I enable the shard
allocation the files are present on 2 nodes, after enabling shard
allocation the 'problem' files are just on one node.
On the nodes with missing files the following WARN message is generated:
[2014-12-31 13:46:48.068 GMT] WARN ||||||
elasticsearch[node2][generic][T#4] org.elasticsearch.indices.cluster
[node2] [index2][0] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[index2][0] failed recovery
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:185)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.index.engine.EngineCreationFailureException:
[index2][0] failed to open reader on writer
at
org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:321)
at
org.elasticsearch.index.shard.service.InternalIndexShard.performRecoveryPrepareForTranslog(InternalIndexShard.java:732)
at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:231)
at
org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
... 3 more
Caused by: java.io.FileNotFoundException: No such file [_ovi.tis]
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:176)
at
org.elasticsearch.index.store.DistributorDirectory.getDirectory(DistributorDirectory.java:144)
at
org.elasticsearch.index.store.DistributorDirectory.openInput(DistributorDirectory.java:130)
at
org.apache.lucene.store.FilterDirectory.openInput(FilterDirectory.java:80)
at
org.elasticsearch.index.store.Store$StoreDirectory.openInput(Store.java:487)
at
org.apache.lucene.codecs.lucene3x.TermInfosReader.(TermInfosReader.java:115)
at
org.apache.lucene.codecs.lucene3x.Lucene3xFields.newTermInfosReader(Lucene3xFields.java:142)
at
org.apache.lucene.codecs.lucene3x.Lucene3xFields.(Lucene3xFields.java:88)
at
org.apache.lucene.codecs.lucene3x.Lucene3xPostingsFormat.fieldsProducer(Lucene3xPostingsFormat.java:62)
at
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:120)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:108)
at
org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:144)
at
org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:238)
at
org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:104)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:422)
at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:112)
at org.apache.lucene.search.SearcherManager.(SearcherManager.java:89)
at
org.elasticsearch.index.engine.internal.InternalEngine.buildSearchManager(InternalEngine.java:1527)
at
org.elasticsearch.index.engine.internal.InternalEngine.start(InternalEngine.java:309)
... 6 more
Is there a way to resolve the checksum problem? Any idea why the files are
deleted?
Any help would be greatly appreciated 
Thank you
Georgeta
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1112b66f-c3bc-42fa-bd63-5757b1a6ebb6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
