# Lucene index corruption on nodes restart **URL:** \ **Category:** Elasticsearch\ **Created:** [March 22, 2014, 1:04pm UTC](https://discuss.elastic.co/t/lucene-index-corruption-on-nodes-restart/16544 "2014-03-22T13:04:56Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Andrey\_Perminov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_perminov/32/103587_2.png) [@Andrey\_Perminov](https://discuss.elastic.co/u/Andrey_Perminov)\ **Post date:** [March 22, 2014, 1:04pm UTC](https://discuss.elastic.co/t/lucene-index-corruption-on-nodes-restart/16544/1 "2014-03-22T13:04:56Z")
We are using a small elasticsearch cluster of three nodes, version 1.0.1. Each node has 7 GB RAM. Our software creates daily indexes for storing it's data. Daily index is something around 5 GB. Unfortunately, for a reason, Elasticsearch eats up all RAM and hangs the node, even though heap size is set to 6 GB max. So we decided to use monit to restart it on reaching memory limit of 90%. It works, but sometimes we got such errors: [2014-03-22 16:56:04,943][DEBUG][action.search.type] [es-00] [product-22-03-2014][0], node[jbUDVzuvS5GTM7iOG8iwzQ], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@687dc039] org.elasticsearch.search.fetch.FetchPhaseExecutionException: [product-22-03-2014][0]: query[filtered(ToParentBlockJoinQuery (filtered(history.created:[1392574921000 TO \*])-\>cache(\_type:\_\_history)))-\>cache(\_type:product)],from[0],size[1000],sort[\]: Fetch Failed [Failed to fetch doc id [7263214]] at org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:230) at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:156) at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:332) at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:304) at org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:71) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4.run(TransportSearchTypeAction.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: java.io.EOFException: seek past EOF: MMapIndexInput(path="/opt/elasticsearch/main/nodes/0/indices/product-22-03-2014/0/index/\_9lz.fdt") at org.apache.lucene.store.ByteBufferIndexInput.seek(ByteBufferIndexInput.java:174) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:229) at org.apache.lucene.index.SegmentReader.document(SegmentReader.java:276) at org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110) at org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196) at org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:228) ... 9 more [2014-03-22 16:56:04,944][DEBUG][action.search.type] [es-00] All shards failed for phase: [query\_fetch] According to our logs, this might happen when one or two nodes get restarted. More strangely, same shard got corrupted on all nodes of cluster. Why could this happen? How can we fix it? Can you suggest us how to fix memory usage? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9e40c2a4-6a76-454d-a96b-483cdbf3e946%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9e40c2a4-6a76-454d-a96b-483cdbf3e946%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andrey\_Perminov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrey_perminov/32/103587_2.png) [@Andrey\_Perminov](https://discuss.elastic.co/u/Andrey_Perminov)\ **Post date:** [March 22, 2014, 1:11pm UTC](https://discuss.elastic.co/t/lucene-index-corruption-on-nodes-restart/16544/2 "2014-03-22T13:11:54Z")
Replication settings are: 1 shard per index, 2 replicas. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c208269d-3d8d-4181-b702-a410a409371a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c208269d-3d8d-4181-b702-a410a409371a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Binh\_Ly\_2](https://avatars.discourse-cdn.com/v4/letter/b/d07c76/32.png) [@Binh\_Ly\_2](https://discuss.elastic.co/u/Binh_Ly_2)\ **Post date:** [March 24, 2014, 4:42pm UTC](https://discuss.elastic.co/t/lucene-index-corruption-on-nodes-restart/16544/3 "2014-03-24T16:42:42Z")
It "sounds" like you might be reaching the RAM limits of your hardware based on your searches. I'd probably try to analyze your searches and (re)size your hardware appropriately. First thing to check is how much fielddata you are using for typical daily queries: curl "localhost:9200/\_nodes/stats/indices/fielddata/\*?pretty" 6GB allocated to ES on a 7GB RAM is probably a bit too much - need some breathing room for OS and file system. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/dfbd6e9b-7278-4eef-aaa4-ca0fa70e4ea5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/dfbd6e9b-7278-4eef-aaa4-ca0fa70e4ea5%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [March 27, 2014, 1:52pm UTC](https://discuss.elastic.co/t/lucene-index-corruption-on-nodes-restart/16544/4 "2014-03-27T13:52:17Z")
Regarding the corruption - that is a lucene problem that we are working on to address. The corruption might have happened a while ago already but you are only happen to see it now. There is no integrity checks in lucene yet so if there is a corrputed index due to HW failure you will also replicate it if you recover from one of those corrputed shards. It's hard to address on the Elasticsaerch level at this point but we will have to fix it in lucene in the very near future. On Saturday, March 22, 2014 2:04:56 PM UTC+1, Andrey Perminov wrote: > We are using a small elasticsearch cluster of three nodes, version 1.0.1. > Each node has 7 GB RAM. Our software creates daily indexes for storing it's > data. Daily index is something around 5 GB. Unfortunately, for a reason, > Elasticsearch eats up all RAM and hangs the node, even though heap size is > set to 6 GB max. So we decided to use monit to restart it on reaching > memory limit of 90%. It works, but sometimes we got such errors: > > [2014-03-22 16:56:04,943][DEBUG][action.search.type] [es-00] > [product-22-03-2014][0], node[jbUDVzuvS5GTM7iOG8iwzQ], [P], s[STARTED]: > Failed to execute [org.elasticsearch.action.search.SearchRequest@687dc039] > org.elasticsearch.search.fetch.FetchPhaseExecutionException: > [product-22-03-2014][0]: query[filtered(ToParentBlockJoinQuery > (filtered(history.created:[1392574921000 TO > \*])-\>cache(\_type:\_\_history)))-\>cache(\_type:product)],from[0],size[1000],sort[\ org.elasticsearch.index.search.nested.NestedFieldComparatorSource@15e4ece9\>]: > Fetch Failed [Failed to fetch doc id [7263214]] > at > org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:230) > at > org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:156) > at > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:332) > at > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:304) > at > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:71) > at > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) > at > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4.run(TransportSearchTypeAction.java:292) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.EOFException: seek past EOF: > MMapIndexInput(path="/opt/elasticsearch/main/nodes/0/indices/product-22-03-2014/0/index/\_9lz.fdt") > at > org.apache.lucene.store.ByteBufferIndexInput.seek(ByteBufferIndexInput.java:174) > at > org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:229) > at > org.apache.lucene.index.SegmentReader.document(SegmentReader.java:276) > at > org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110) > at > org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196) > at > org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:228) > ... 9 more > [2014-03-22 16:56:04,944][DEBUG][action.search.type] [es-00] All > shards failed for phase: [query\_fetch] > > According to our logs, this might happen when one or two nodes get > restarted. More strangely, same shard got corrupted on all nodes of > cluster. Why could this happen? How can we fix it? Can you suggest us how > to fix memory usage? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/add5f603-4088-4344-ac1d-1accfc7a487b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/add5f603-4088-4344-ac1d-1accfc7a487b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Pawel\_Chabierski](https://avatars.discourse-cdn.com/v4/letter/p/ecc23a/32.png) [@Pawel\_Chabierski](https://discuss.elastic.co/u/Pawel_Chabierski)\ **Post date:** [April 2, 2014, 9:23am UTC](https://discuss.elastic.co/t/lucene-index-corruption-on-nodes-restart/16544/5 "2014-04-02T09:23:38Z")
Few days ago we found we've got that same error when we search for data. reason: "FetchPhaseExecutionException[[site\_production][1]: query[ConstantScore(cache(\_type:ademail))],from[0],size[648]: Fetch Failed [Failed to fetch doc id [9615533]]]; nested: EOFException[seek past EOF: MMapIndexInput(path="/opt/elasticsearch/data/naprawa/nodes/0/indices/site\_production/1/index/\_573oa.fdt")]; After this error, elasticsearch stop search and don't return all results, even if they are returned properly in other query. Is there any way to fix this index? We try checkIndex from Luccene core library, after that we lost ~20 milions of documents but error still occurs. Also we try to restore index from snapshot, but error still occurs :/. W dniu sobota, 22 marca 2014 14:04:56 UTC+1 użytkownik Andrey Perminov napisał: > We are using a small elasticsearch cluster of three nodes, version 1.0.1. > Each node has 7 GB RAM. Our software creates daily indexes for storing it's > data. Daily index is something around 5 GB. Unfortunately, for a reason, > Elasticsearch eats up all RAM and hangs the node, even though heap size is > set to 6 GB max. So we decided to use monit to restart it on reaching > memory limit of 90%. It works, but sometimes we got such errors: > > [2014-03-22 16:56:04,943][DEBUG][action.search.type] [es-00] > [product-22-03-2014][0], node[jbUDVzuvS5GTM7iOG8iwzQ], [P], s[STARTED]: > Failed to execute [org.elasticsearch.action.search.SearchRequest@687dc039] > org.elasticsearch.search.fetch.FetchPhaseExecutionException: > [product-22-03-2014][0]: query[filtered(ToParentBlockJoinQuery > (filtered(history.created:[1392574921000 TO > \*])-\>cache(\_type:\_\_history)))-\>cache(\_type:product)],from[0],size[1000],sort[\ org.elasticsearch.index.search.nested.NestedFieldComparatorSource@15e4ece9\>]: > Fetch Failed [Failed to fetch doc id [7263214]] > at > org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:230) > at > org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:156) > at > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:332) > at > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:304) > at > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:71) > at > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) > at > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4.run(TransportSearchTypeAction.java:292) > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > Source) > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > Source) > at java.lang.Thread.run(Unknown Source) > Caused by: java.io.EOFException: seek past EOF: > MMapIndexInput(path="/opt/elasticsearch/main/nodes/0/indices/product-22-03-2014/0/index/\_9lz.fdt") > at > org.apache.lucene.store.ByteBufferIndexInput.seek(ByteBufferIndexInput.java:174) > at > org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:229) > at > org.apache.lucene.index.SegmentReader.document(SegmentReader.java:276) > at > org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110) > at > org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196) > at > org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:228) > ... 9 more > [2014-03-22 16:56:04,944][DEBUG][action.search.type] [es-00] All > shards failed for phase: [query\_fetch] > > According to our logs, this might happen when one or two nodes get > restarted. More strangely, same shard got corrupted on all nodes of > cluster. Why could this happen? How can we fix it? Can you suggest us how > to fix memory usage? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8831e9e9-9c4d-4a2e-bd16-99c33f589b7f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8831e9e9-9c4d-4a2e-bd16-99c33f589b7f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [April 2, 2014, 12:01pm UTC](https://discuss.elastic.co/t/lucene-index-corruption-on-nodes-restart/16544/6 "2014-04-02T12:01:21Z")
hey, is it possible to look at this index / shard? do you still have it / can you safe it for further investigations? You can ping me directly at simon AT elasticsearch DOT com On Wednesday, April 2, 2014 11:23:38 AM UTC+2, Paweł Chabierski wrote: > Few days ago we found we've got that same error when we search for data. > > reason: "FetchPhaseExecutionException[[site\_production][1]: > query[ConstantScore(cache(\_type:ademail))],from[0],size[648]: Fetch Failed > [Failed to fetch doc id [9615533]]]; nested: EOFException[seek past EOF: > MMapIndexInput(path="/opt/elasticsearch/data/naprawa/nodes/0/indices/site\_production/1/index/\_573oa.fdt")]; > > After this error, elasticsearch stop search and don't return all results, > even if they are returned properly in other query. Is there any way to fix > this index? We try checkIndex from Luccene core library, after that we lost > ~20 milions of documents but error still occurs. Also we try to restore > index from snapshot, but error still occurs :/. > > W dniu sobota, 22 marca 2014 14:04:56 UTC+1 użytkownik Andrey Perminov > napisał: > > > We are using a small elasticsearch cluster of three nodes, version 1.0.1. > > Each node has 7 GB RAM. Our software creates daily indexes for storing it's > > data. Daily index is something around 5 GB. Unfortunately, for a reason, > > Elasticsearch eats up all RAM and hangs the node, even though heap size is > > set to 6 GB max. So we decided to use monit to restart it on reaching > > memory limit of 90%. It works, but sometimes we got such errors: > > > > [2014-03-22 16:56:04,943][DEBUG][action.search.type] [es-00] > > [product-22-03-2014][0], node[jbUDVzuvS5GTM7iOG8iwzQ], [P], s[STARTED]: > > Failed to execute [org.elasticsearch.action.search.SearchRequest@687dc039] > > org.elasticsearch.search.fetch.FetchPhaseExecutionException: > > [product-22-03-2014][0]: query[filtered(ToParentBlockJoinQuery > > (filtered(history.created:[1392574921000 TO > > \*])-\>cache(\_type:\_\_history)))-\>cache(\_type:product)],from[0],size[1000],sort[\ > org.elasticsearch.index.search.nested.NestedFieldComparatorSource@15e4ece9\>]: > > Fetch Failed [Failed to fetch doc id [7263214]] > > at > > org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:230) > > at > > org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:156) > > at > > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:332) > > at > > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:304) > > at > > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:71) > > at > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) > > at > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4.run(TransportSearchTypeAction.java:292) > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > Source) > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > Source) > > at java.lang.Thread.run(Unknown Source) > > Caused by: java.io.EOFException: seek past EOF: > > MMapIndexInput(path="/opt/elasticsearch/main/nodes/0/indices/product-22-03-2014/0/index/\_9lz.fdt") > > at > > org.apache.lucene.store.ByteBufferIndexInput.seek(ByteBufferIndexInput.java:174) > > at > > org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:229) > > at > > org.apache.lucene.index.SegmentReader.document(SegmentReader.java:276) > > at > > org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110) > > at > > org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196) > > at > > org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:228) > > ... 9 more > > [2014-03-22 16:56:04,944][DEBUG][action.search.type] [es-00] All > > shards failed for phase: [query\_fetch] > > > > According to our logs, this might happen when one or two nodes get > > restarted. More strangely, same shard got corrupted on all nodes of > > cluster. Why could this happen? How can we fix it? Can you suggest us how > > to fix memory usage? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b9d9aa67-5adc-4c06-8659-9031cb673d39%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b9d9aa67-5adc-4c06-8659-9031cb673d39%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![slagraulet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slagraulet/32/1481_2.png) [@slagraulet](https://discuss.elastic.co/u/slagraulet)\ **Post date:** [June 18, 2014, 1:55pm UTC](https://discuss.elastic.co/t/lucene-index-corruption-on-nodes-restart/16544/7 "2014-06-18T13:55:51Z")
Hi Simon, I think we hit the same problem on our index. I have one shard which sometimes give me this answer: "\_shards": { "total": 5, "successful": 4, "failed": 1, "failures": [ { "index": "products", "shard": 0, "status": 500, "reason": "QueryPhaseExecutionException[[products][0]: query[filtered(filtered(id:finlandek short\_title:finlandek title:finlandek category\_name:finlandek description:finlandek category\_name\_path:finlandek brand\_name:finlandek)-\>cache(stats\_income7\_global:[0 TO \*]))-\>cache(org.elasticsearch.index.search.nested.NonNestedDocsFilter@3977d235)],from[0],size[10]: Query Failed [Failed to execute main query]]; nested: EOFException[seek past EOF: MMapIndexInput(path="/data/es/elasticsearch-1.3.0-SNAPSHOT/data/elasticsearch/nodes/0/indices/products/0/index/\_za6\_es090\_0.doc")]; " } ] }, Is it still under investigation? Le mercredi 2 avril 2014 14:01:21 UTC+2, simonw a écrit : > hey, > > is it possible to look at this index / shard? do you still have it / can > you safe it for further investigations? You can ping me directly at simon > AT elasticsearch DOT com > > On Wednesday, April 2, 2014 11:23:38 AM UTC+2, Paweł Chabierski wrote: > > > Few days ago we found we've got that same error when we search for data. > > > > reason: "FetchPhaseExecutionException[[site\_production][1]: > > query[ConstantScore(cache(\_type:ademail))],from[0],size[648]: Fetch Failed > > [Failed to fetch doc id [9615533]]]; nested: EOFException[seek past EOF: > > MMapIndexInput(path="/opt/elasticsearch/data/naprawa/nodes/0/indices/site\_production/1/index/\_573oa.fdt")]; > > > > After this error, elasticsearch stop search and don't return all results, > > even if they are returned properly in other query. Is there any way to fix > > this index? We try checkIndex from Luccene core library, after that we lost > > ~20 milions of documents but error still occurs. Also we try to restore > > index from snapshot, but error still occurs :/. > > > > W dniu sobota, 22 marca 2014 14:04:56 UTC+1 użytkownik Andrey Perminov > > napisał: > > > > > We are using a small elasticsearch cluster of three nodes, version > > > 1.0.1. Each node has 7 GB RAM. Our software creates daily indexes for > > > storing it's data. Daily index is something around 5 GB. Unfortunately, for > > > a reason, Elasticsearch eats up all RAM and hangs the node, even though > > > heap size is set to 6 GB max. So we decided to use monit to restart it on > > > reaching memory limit of 90%. It works, but sometimes we got such errors: > > > > > > [2014-03-22 16:56:04,943][DEBUG][action.search.type] [es-00] > > > [product-22-03-2014][0], node[jbUDVzuvS5GTM7iOG8iwzQ], [P], s[STARTED]: > > > Failed to execute [org.elasticsearch.action.search.SearchRequest@687dc039] > > > org.elasticsearch.search.fetch.FetchPhaseExecutionException: > > > [product-22-03-2014][0]: query[filtered(ToParentBlockJoinQuery > > > (filtered(history.created:[1392574921000 TO > > > \*])-\>cache(\_type:\_\_history)))-\>cache(\_type:product)],from[0],size[1000],sort[\ > > org.elasticsearch.index.search.nested.NestedFieldComparatorSource@15e4ece9\>]: > > > Fetch Failed [Failed to fetch doc id [7263214]] > > > at > > > org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:230) > > > at > > > org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:156) > > > at > > > org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:332) > > > at > > > org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteFetch(SearchServiceTransportAction.java:304) > > > at > > > org.elasticsearch.action.search.type.TransportSearchQueryAndFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryAndFetchAction.java:71) > > > at > > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:216) > > > at > > > org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$4.run(TransportSearchTypeAction.java:292) > > > at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown > > > Source) > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown > > > Source) > > > at java.lang.Thread.run(Unknown Source) > > > Caused by: java.io.EOFException: seek past EOF: > > > MMapIndexInput(path="/opt/elasticsearch/main/nodes/0/indices/product-22-03-2014/0/index/\_9lz.fdt") > > > at > > > org.apache.lucene.store.ByteBufferIndexInput.seek(ByteBufferIndexInput.java:174) > > > at > > > org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:229) > > > at > > > org.apache.lucene.index.SegmentReader.document(SegmentReader.java:276) > > > at > > > org.apache.lucene.index.BaseCompositeReader.document(BaseCompositeReader.java:110) > > > at > > > org.apache.lucene.search.IndexSearcher.doc(IndexSearcher.java:196) > > > at > > > org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:228) > > > ... 9 more > > > [2014-03-22 16:56:04,944][DEBUG][action.search.type] [es-00] All > > > shards failed for phase: [query\_fetch] > > > > > > According to our logs, this might happen when one or two nodes get > > > restarted. More strangely, same shard got corrupted on all nodes of > > > cluster. Why could this happen? How can we fix it? Can you suggest us how > > > to fix memory usage? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/05ea6cde-87f2-4c0d-ac99-7d6aa769383c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/05ea6cde-87f2-4c0d-ac99-7d6aa769383c%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:21am UTC](https://discuss.elastic.co/t/lucene-index-corruption-on-nodes-restart/16544/8 "2017-07-06T01:21:31Z")