Corrupt index, checksum failed

Sviatoslav_Abakumov · August 29, 2014, 5:58am

Hello.

I was using Elasticsearch v1.2.2 on Ubuntu 14.04 x64 with SSD. I have one
index 'myindex' with around 100k of not so complex documents plus indices
that Marvel generates.

Recently I started getting such kind of errors:

gist.github.com

https://gist.github.com/Perlence/51aae04d077a53919c31

elasticsearch.log

[2014-08-29 05:30:07,112][WARN ][cluster.action.shard     ] [Morning Star] [myindex][9] sending failed shard for [myindex][9], node[bhRsAtMtR9iFN8VNB0zo8Q], [R], s[INITIALIZING], indexUUID [Aww2M7B7RLS5WVBLJE2E2g], reason [engine failure, message [corrupted preexisting index][CorruptIndexException[[myindex][9] Corrupted index [corrupted_n85SCtYYTsm4Eux7rjPxtg] caused by: CorruptIndexException[checksum failed (hardware problem?) : expected=r95zlm actual=1yl70z3 (resource=name [_29_es090_0.tim], length [403211], checksum [r95zlm], writtenBy [LUCENE_4_9])]]]]
[2014-08-29 05:30:07,148][WARN ][indices.cluster          ] [Morning Star] [myindex][4] failed to start shard
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [myindex][4] failed to fetch index version after copying it over
        at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:152)
        at org.elasticsearch.index.gateway.IndexShardGatewayService$1.run(IndexShardGatewayService.java:132)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.lucene.index.CorruptIndexException: [myindex][4] Corrupted index [corrupted_PCX3Uc0vTrOc-G4xDCOT9g] caused by: CorruptIndexException[checksum failed (hardware problem?) : expected=82a20f8 actual=315958b0 (resource=BufferedChecksumIndexInput(NIOFSIndexInput(path="/var/lib/elasticsearch/mycluster/nodes/0/indices/myindex/4/index/_54.fdt")))]
        at org.elasticsearch.index.store.Store.failIfCorrupted(Store.java:343)

This file has been truncated. show original

It looks like shards become unassigned after some time and fail to assign
again.

I've taken these steps:

set index.routing.allocation.disable_allocation to false
increased heap size from 1 GB to 16 GB
increased number of shards
individual insert instead of of bulk insert
updated Elasticsearch from 1.2.2 to 1.3.2
started two instances of ES with 8 GB of heap each and 1 replica one
the same machine
deleted index
provided gateway.recover_after_nodes and gateway.expected_nodes
settings in config
reduced heap size to 4 GB per node

I've also noticed that shards of Marvel's indices become unassigned too.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4fb025ee-3668-4539-b249-11bcba216ea0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.