# Shard index gone bad, anyone know how to fix this: java.io.EOFException: read past EOF: NIOFSIndexInput **URL:** \ **Category:** Elasticsearch\ **Created:** [January 4, 2013, 9:36am UTC](https://discuss.elastic.co/t/shard-index-gone-bad-anyone-know-how-to-fix-this-java-io-eofexception-read-past-eof-niofsindexinput/10227 "2013-01-04T09:36:36Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Dimitry](https://avatars.discourse-cdn.com/v4/letter/d/838e76/32.png) [@Dimitry](https://discuss.elastic.co/u/Dimitry)\ **Post date:** [January 4, 2013, 9:36am UTC](https://discuss.elastic.co/t/shard-index-gone-bad-anyone-know-how-to-fix-this-java-io-eofexception-read-past-eof-niofsindexinput/10227/1 "2013-01-04T09:36:36Z")
We running version 0.19.9 with 6 servers running using 6 shards. A few days ago, shard number 2 seems to have gone goofy (looks like a corruption in the index) causing the following exception to appear constantly in the server logs: _org.elasticsearch.transport.RemoteTransportException: [cardano][inet[/xx.xxx.xx.xxx:9300]][search/phase/query]_ _Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: [theindex][2]: query[filtered(+activityObject.content:"Some query term" +sourceInfo.publisher:Some Name -sourceInfo.dataSource:directPooling)-\>cache(\_type:socialmedia)],from[0],size[1],sort[\!]: Query Failed [Failed to execute main query]_ - at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:182)\* - at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234) - - at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:497) - - at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:486) - - at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268) - - at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) - - at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) - - at java.lang.Thread.run(Thread.java:662)\* \*Caused by: java.io.EOFException: read past EOF: NIOFSIndexInput(path="/var/data/elasticsearch/nodes/0/indices/theindex/2/index/\_161lvl.tis") - - at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:264) - - at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:40) - - at org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)\* - at org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:217) - What I've tried: - Changed replication to 0 - Closed/Opened the index (to force rebalancing) - Restarted the node containing shard 2 with index.shard.check\_on\_startup: true This seems to point to something going bad with the lucene index of this shard, you the check\_on\_startup didn't seem to solve the problem. Anyone know how to get around this. Much appreciated. Dimitry. -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 5, 2013, 4:31pm UTC](https://discuss.elastic.co/t/shard-index-gone-bad-anyone-know-how-to-fix-this-java-io-eofexception-read-past-eof-niofsindexinput/10227/2 "2013-01-05T16:31:17Z")
Hi Dimitri, Read past EOF? I never got that. But here's my braindump, nevertheless: If there's a problem with shard 2 on all nodes, I don't know what you can do to recover the data, other than reindex or restore from backup. If you get this issue only on one server, then I'd try something like this: - shut down the problematic node - reduce the number of replicas by 1 - move /var/data/elasticsearch/nodes/0/indices/theindex/2/ to some backup location - start the node again - increase the number of replicas back to force replication I would assume that if this doesn't fix it it's either: - a problem with the hardware on that node. You can check memory&hdd, or try to reproduce with another machine to confirm/deny - a problem with shard 2 on that index across all the nodes, in which case you'd be back to reindexing/restoring from backup ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Fri, Jan 4, 2013 at 11:36 AM, Dimitry [dlvovsky@gmail.com](mailto:dlvovsky@gmail.com) wrote: > We running version 0.19.9 with 6 servers running using 6 shards. A few > days ago, shard number 2 seems to have gone goofy (looks like > a corruption in the index) causing the following exception to appear > constantly in the server logs: > > _org.elasticsearch.transport.RemoteTransportException: > [cardano][inet[/xx.xxx.xx.xxx:9300]][search/phase/query]_ > _Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: > [theindex][2]: query[filtered(+activityObject.content:"Some query term" > +sourceInfo.publisher:Some Name > -sourceInfo.dataSource:directPooling)-\>cache(\_type:socialmedia)],from[0],size[1],sort[\ org.elasticsearch.index.field.data.longs.LongFieldDataType$1@24f99e97\>!]: > Query Failed [Failed to execute main query]_ > > - at > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:182)\* > - at > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234) > - > - at > org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:497) > - > - at > org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:486) > - > - at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268) > - > - at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > - > - at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > - > - at java.lang.Thread.run(Thread.java:662)\* > \*Caused by: java.io.EOFException: read past EOF: > NIOFSIndexInput(path="/var/data/elasticsearch/nodes/0/indices/theindex/2/index/\_161lvl.tis") > - > - at > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:264) > - > - at > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:40) > - > - at org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)\* > - at > org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:217) > - > > What I've tried: > > - Changed replication to 0 > - Closed/Opened the index (to force rebalancing) > - Restarted the node containing shard 2 > with index.shard.check\_on\_startup: true > > This seems to point to something going bad with the lucene index of this > shard, you the check\_on\_startup didn't seem to solve the problem. Anyone > know how to get around this. > > Much appreciated. > Dimitry. > > -- -- ---
**Author:** ![Dimitry](https://avatars.discourse-cdn.com/v4/letter/d/838e76/32.png) [@Dimitry](https://discuss.elastic.co/u/Dimitry)\ **Post date:** [January 29, 2013, 11:46am UTC](https://discuss.elastic.co/t/shard-index-gone-bad-anyone-know-how-to-fix-this-java-io-eofexception-read-past-eof-niofsindexinput/10227/3 "2013-01-29T11:46:58Z")
Running java -cp lucene-core-3.6.1.jar -ea:org.apache.lucene... org.apache.lucene.index.CheckIndex /home/es/data/production/nodes/0/indices/users/1/index/ -fix did the trick as reported in this post [https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/xprRlA8RQ90](https://groups.google.com/forum/?fromgroups=#!topic/elasticsearch/xprRlA8RQ90) by Marcin Dojwa. Thanks to all. On Saturday, January 5, 2013 5:31:17 PM UTC+1, Radu Gheorghe wrote: > Hi Dimitri, > > Read past EOF? I never got that. But here's my braindump, nevertheless: > > If there's a problem with shard 2 on all nodes, I don't know what you can > do to recover the data, other than reindex or restore from backup. > > If you get this issue only on one server, then I'd try something like this: > > - shut down the problematic node > - reduce the number of replicas by 1 > - move /var/data/elasticsearch/nodes/0/indices/theindex/2/ to some backup > location > - start the node again > - increase the number of replicas back to force replication > > I would assume that if this doesn't fix it it's either: > > - a problem with the hardware on that node. You can check memory&hdd, or > try to reproduce with another machine to confirm/deny > - a problem with shard 2 on that index across all the nodes, in which case > you'd be back to reindexing/restoring from backup > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Fri, Jan 4, 2013 at 11:36 AM, Dimitry \<[dlvo...@gmail.com](mailto:dlvo...@gmail.com) \\>wrote: > > > We running version 0.19.9 with 6 servers running using 6 shards. A few > > days ago, shard number 2 seems to have gone goofy (looks like > > a corruption in the index) causing the following exception to appear > > constantly in the server logs: > > > > _org.elasticsearch.transport.RemoteTransportException: > > [cardano][inet[/xx.xxx.xx.xxx:9300]][search/phase/query]_ > > _Caused by: org.elasticsearch.search.query.QueryPhaseExecutionException: > > [theindex][2]: query[filtered(+activityObject.content:"Some query term" > > +sourceInfo.publisher:Some Name > > -sourceInfo.dataSource:directPooling)-\>cache(\_type:socialmedia)],from[0],size[1],sort[\ > org.elasticsearch.index.field.data.longs.LongFieldDataType$1@24f99e97\>!]: > > Query Failed [Failed to execute main query]_ > > > > - at > > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:182)\* > > - at > > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:234) > > - > > - at > > org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:497) > > - > > - at > > org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:486) > > - > > - at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:268) > > - > > - at > > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > > - > > - at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > > - > > - at java.lang.Thread.run(Thread.java:662)\* > > \*Caused by: java.io.EOFException: read past EOF: > > NIOFSIndexInput(path="/var/data/elasticsearch/nodes/0/indices/theindex/2/index/\_161lvl.tis") > > - > > - at > > org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:264) > > - > > - at > > org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:40) > > - > > - at org.apache.lucene.store.DataInput.readVInt(DataInput.java:107)\* > > - at > > org.apache.lucene.store.BufferedIndexInput.readVInt(BufferedIndexInput.java:217) > > - > > > > What I've tried: > > > > - Changed replication to 0 > > - Closed/Opened the index (to force rebalancing) > > - Restarted the node containing shard 2 > > with index.shard.check\_on\_startup: true > > > > This seems to point to something going bad with the lucene index of this > > shard, you the check\_on\_startup didn't seem to solve the problem. Anyone > > know how to get around this. > > > > Much appreciated. > > Dimitry. > > > > -- -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:54am UTC](https://discuss.elastic.co/t/shard-index-gone-bad-anyone-know-how-to-fix-this-java-io-eofexception-read-past-eof-niofsindexinput/10227/4 "2017-07-06T02:54:13Z")