# Found unrecoverable error for ES 6.0 **URL:** \ **Category:** Elasticsearch\ **Created:** [November 28, 2017, 8:06am UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331 "2017-11-28T08:06:51Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![winsen684](https://avatars.discourse-cdn.com/v4/letter/w/258eb7/32.png) [@winsen684](https://discuss.elastic.co/u/winsen684)\ **Post date:** [November 28, 2017, 8:06am UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331/1 "2017-11-28T08:06:52Z")
Log: 17/11/28 13:52:52 ERROR TaskContextImpl: Error in TaskCompletionListener org.elasticsearch.hadoop.rest.EsHadoopInvalidRequest: Found unrecoverable error [x.x.x.x:9205] returned Internal Server Error(500) - compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/data1/elasticsearch/data/nodes/0/indices/S93P5ab\_S42YDhmslWDKAQ/33/index/\_9qa\_Lucene50\_0.doc"))); Bailing out.. at org.elasticsearch.hadoop.rest.RestClient.processBulkResponse(RestClient.java:251) at org.elasticsearch.hadoop.rest.RestClient.bulk(RestClient.java:203) at org.elasticsearch.hadoop.rest.RestRepository.tryFlush(RestRepository.java:248) at org.elasticsearch.hadoop.rest.RestRepository.flush(RestRepository.java:270) at org.elasticsearch.hadoop.rest.RestRepository.close(RestRepository.java:295) at org.elasticsearch.hadoop.rest.RestService$PartitionWriter.close(RestService.java:121) at org.elasticsearch.spark.rdd.EsRDDWriter$$anonfun$write$1.apply(EsRDDWriter.scala:60) at org.elasticsearch.spark.rdd.EsRDDWriter$$anonfun$write$1.apply(EsRDDWriter.scala:60) at org.apache.spark.TaskContext$$anon$1.onTaskCompletion(TaskContext.scala:123) at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:97) at org.apache.spark.TaskContextImpl$$anonfun$markTaskCompleted$1.apply(TaskContextImpl.scala:95) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:95) at org.apache.spark.scheduler.Task.run(Task.scala:112) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 17/11/28 13:52:52 ERROR Executor: Exception in task 5980.0 in stage 1.0 (TID 7319) org.apache.spark.util.TaskCompletionListenerException: Found unrecoverable error [x.x.x.x:9205] returned Internal Server Error(500) - compound sub-files must have a valid codec header and footer: file is too small (0 bytes) (resource=BufferedChecksumIndexInput(MMapIndexInput(path="/data1/elasticsearch/data/nodes/0/indices/S93P5ab\_S42YDhmslWDKAQ/33/index/\_9qa\_Lucene50\_0.doc"))); Bailing out.. at org.apache.spark.TaskContextImpl.markTaskCompleted(TaskContextImpl.scala:105) at org.apache.spark.scheduler.Task.run(Task.scala:112) at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) My Cluster has 5 SSD Servers with CentOS7. There are 6 instances on every Server. They are belonged to data nodes. Other 3 virtual machine are master nodes, 3 virtual machine are ingest nodes. ---
**Author:** ![winsen684](https://avatars.discourse-cdn.com/v4/letter/w/258eb7/32.png) [@winsen684](https://discuss.elastic.co/u/winsen684)\ **Post date:** [November 30, 2017, 1:48am UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331/2 "2017-11-30T01:48:55Z")
Server logļ¼š [2017-11-29T20:37:40,449][WARN][o.e.c.a.s.ShardStateAction] [host:9300] [auto-index-2017-11-28][3] received shard failed for shard id [[auto-index-2017-11-28][3]], allocation id [midSbxhxTqyKDGwhTPFmng], primary term [0], message [shard failure, reason [refresh failed]], failure [CorruptIndexException[Problem reading index from store(MMapDirectory@/data4/elasticsearch/data/nodes/0/indices/5MSqrobDQ2mSzZzCZGO-5g/3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@cd8c03a) (resource=store(MMapDirectory@/data4/elasticsearch/data/nodes/0/indices/5MSqrobDQ2mSzZzCZGO-5g/3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@cd8c03a))]; nested: EOFException[read past EOF: MMapIndexInput(path="/data4/elasticsearch/data/nodes/0/indices/5MSqrobDQ2mSzZzCZGO-5g/3/index/\_3iu.cfe")]; ] org.apache.lucene.index.CorruptIndexException: Problem reading index from store(MMapDirectory@/data4/elasticsearch/data/nodes/0/indices/5MSqrobDQ2mSzZzCZGO-5g/3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@cd8c03a) (resource=store(MMapDirectory@/data4/elasticsearch/data/nodes/0/indices/5MSqrobDQ2mSzZzCZGO-5g/3/index lockFactory=org.apache.lucene.store.NativeFSLockFactory@cd8c03a)) at org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:140) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.SegmentReader.(SegmentReader.java:78) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.ReadersAndUpdates.getReader(ReadersAndUpdates.java:208) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.ReadersAndUpdates.getReadOnlyClone(ReadersAndUpdates.java:258) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:105) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:490) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:293) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:268) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:258) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.FilterDirectoryReader.doOpenIfChanged(FilterDirectoryReader.java:104) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:140) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:156) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.search.SearcherManager.refreshIfNeeded(SearcherManager.java:58) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.search.ReferenceManager.doMaybeRefresh(ReferenceManager.java:176) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.apache.lucene.search.ReferenceManager.maybeRefreshBlocking(ReferenceManager.java:253) ~[lucene-core-7.0.1.jar:7.0.1 8d6c3889aa543954424d8ac1dbb3f03bf207140b - sarowe - 2017-10-02 14:36:35] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1207) ~[elasticsearch-6.0.0.jar:6.0.0] at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:855) ~[elasticsearch-6.0.0.jar:6.0.0] at org.elasticsearch.index.engine.InternalEngine.refresh(InternalEngine.java:1207) ~[elasticsearch-6.0.0.jar:6.0.0] at org.elasticsearch.index.shard.IndexShard.refresh(IndexShard.java:855) ~[elasticsearch-6.0.0.jar:6.0.0] at org.elasticsearch.index.IndexService.maybeRefreshEngine(IndexService.java:695) ~[elasticsearch-6.0.0.jar:6.0.0] at org.elasticsearch.index.IndexService.access$400(IndexService.java:97) ~[elasticsearch-6.0.0.jar:6.0.0] at org.elasticsearch.index.IndexService$AsyncRefreshTask.runInternal(IndexService.java:899) ~[elasticsearch-6.0.0.jar:6.0.0] at org.elasticsearch.index.IndexService$BaseAsyncTask.run(IndexService.java:809) ~[elasticsearch-6.0.0.jar:6.0.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:569) ~[elasticsearch-6.0.0.jar:6.0.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0\_152] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0\_152] at java.lang.Thread.run(Thread.java:748) [?:1.8.0\_152] Caused by: java.io.EOFException: read past EOF: MMapIndexInput(path="/data4/elasticsearch/data/nodes/0/indices/5MSqrobDQ2mSzZzCZGO-5g/3/index/\_3iu.cfe") It seems ES has a serious bug. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 1, 2017, 8:28am UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331/3 "2017-12-01T08:28:30Z")
can you check the logfiles of the affected node for any exceptions/corruptions. It looks to me as if some data on there is corrupted, because some file is of size 0, as mentioned at the top. ---
**Author:** ![winsen684](https://avatars.discourse-cdn.com/v4/letter/w/258eb7/32.png) [@winsen684](https://discuss.elastic.co/u/winsen684)\ **Post date:** [December 1, 2017, 12:31pm UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331/4 "2017-12-01T12:31:39Z")
I find a work round. When file system is ext4 or xfs, this issue must happen. But I change it to btrfs, there is no this issue. So I think it's relative with ssd, file system and os. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 1, 2017, 1:42pm UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331/5 "2017-12-01T13:42:25Z")
I think the majority of users runs on ext4 or xfs instead of brtfs. Are you running any special linux distribution? Also, is this an NFS volume? Curious about your setup... ---
**Author:** ![winsen684](https://avatars.discourse-cdn.com/v4/letter/w/258eb7/32.png) [@winsen684](https://discuss.elastic.co/u/winsen684)\ **Post date:** [December 4, 2017, 4:32am UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331/6 "2017-12-04T04:32:38Z")
The os is "CentOS Linux release 7.2.1511 (Core)". After weekend running, btrfs also has issue. The shards become unassign. They are not NFS volume. No clue for this issue now. My setup consist of 3 master, 2 ingest, 30 data. 30 data are deployed on 5 ssd machines. All roles are running on CentOS7. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [December 4, 2017, 8:16am UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331/7 "2017-12-04T08:16:30Z")
have you tried running on another physical harddisk on the same node, and check if that keeps happening - I'd slowly consider a hardware failure here, unless there is some fancy script that nulls out your files... have you checked dmesg output that might indicate a hardware failure? ---
**Author:** ![winsen684](https://avatars.discourse-cdn.com/v4/letter/w/258eb7/32.png) [@winsen684](https://discuss.elastic.co/u/winsen684)\ **Post date:** [December 4, 2017, 9:10am UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331/8 "2017-12-04T09:10:12Z")
Yes, 1 SSD machine has issue, then 1 node is disconnected. So btrfs is ok for es 6.0. I will continue see it after now. Thank you! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 1, 2018, 9:22am UTC](https://discuss.elastic.co/t/found-unrecoverable-error-for-es-6-0/109331/9 "2018-01-01T09:22:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.