# Java.io.IOException: Input/output error: NIOFSIndexInput **URL:** \ **Category:** Elasticsearch\ **Created:** [September 2, 2013, 4:07am UTC](https://discuss.elastic.co/t/java-io-ioexception-input-output-error-niofsindexinput/13426 "2013-09-02T04:07:54Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [September 2, 2013, 4:07am UTC](https://discuss.elastic.co/t/java-io-ioexception-input-output-error-niofsindexinput/13426/1 "2013-09-02T04:07:54Z")
Hello, recently we receive a lot of these error message in our elasticsearch nodes.Initially, we think that might be due to hardware issue and we replace the disk but we are still seeing this issue now and then. It occur more frequently lately. Should we worry on this exception? What actually is happening and how can we prevent this from happening? A little background of our environment, using elasticsearch version 0.20.5 and 9 gc nodes in total. [2013-08-30 15:49:45,794][WARN][index.merge.scheduler] [node1] [MyIndex][57] failed to merge java.io.IOException: Input/output error: NIOFSIndexInput(path="/vol/elasticsearch/MyIndex\_Cluster/nodes/0/indices/MyIndex/57/index/\_12sww.fdx") at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:180) at org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:270) at org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:40) at org.apache.lucene.store.DataInput.readInt(DataInput.java:86) at org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:179) at org.apache.lucene.store.DataInput.readLong(DataInput.java:130) at org.apache.lucene.store.BufferedIndexInput.readLong(BufferedIndexInput.java:192) at org.apache.lucene.index.FieldsReader.rawDocs(FieldsReader.java:292) at org.apache.lucene.index.SegmentMerger.copyFieldsWithDeletions(SegmentMerger.java:265) at org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:223) at org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:107) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4263) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388) at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456) Caused by: java.io.IOException: Input/output error at sun.nio.ch.FileDispatcher.pread0(Native Method) at sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:31) at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:199) at sun.nio.ch.IOUtil.read(IOUtil.java:175) at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:612) at org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:162) ... 15 more -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 2, 2013, 7:34am UTC](https://discuss.elastic.co/t/java-io-ioexception-input-output-error-niofsindexinput/13426/2 "2013-09-02T07:34:41Z")
You have a RAM issue. Input of data fails because your RAM for I/O is too small, while segment merging is going on. Consider decreasing the maximum segment size for merging (default is 5g, my recommendation is 2g) Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [September 2, 2013, 9:17am UTC](https://discuss.elastic.co/t/java-io-ioexception-input-output-error-niofsindexinput/13426/3 "2013-09-02T09:17:52Z")
Hi Jörg, Thank you for your insight. When you said it is RAM issue, I take it as the amount of heap allocated to elasticsearch jvm? Read into merging documentation here, [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/merge/) , we are currently using the default 5g, will increase the jvm heap twice (from 8G to 16G) prevent this issue from happening again? Jason On Monday, September 2, 2013 3:34:41 PM UTC+8, Jörg Prante wrote: > You have a RAM issue. Input of data fails because your RAM for I/O is too > small, while segment merging is going on. > > Consider decreasing the maximum segment size for merging (default is 5g, > my recommendation is 2g) > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 2, 2013, 9:43am UTC](https://discuss.elastic.co/t/java-io-ioexception-input-output-error-niofsindexinput/13426/4 "2013-09-02T09:43:36Z")
If you have enough RAM outside JVM available to increase the heap space, it may help. It depends on your requirements for processing large segment merges. In my view, 5g sized segments are a bit heavy and take also long time to get merged, much CPU and probably also long GC. You should check on disk if you can observe files in the data path if some got as large as ~5g. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [September 2, 2013, 11:18am UTC](https://discuss.elastic.co/t/java-io-ioexception-input-output-error-niofsindexinput/13426/5 "2013-09-02T11:18:02Z")
Hi Jörg, Yes, in the meantime, we have sufficient memory to a single node which each has 32GB memory in total. Notice that the offheap memory usage always hover around 40-60MB.. so the merging should have been done in the jvm heap. Any specific files in in /vol/elasticsearch/MyIndex\_Cluster/nodes/0/indices/MyIndex/[dd] should we monitor in particular or as a collective summation? Jason On Mon, Sep 2, 2013 at 5:43 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \<[joergprante@gmail.com](mailto:joergprante@gmail.com) > wrote: > If you have enough RAM outside JVM available to increase the heap space, > it may help. It depends on your requirements for processing large segment > merges. In my view, 5g sized segments are a bit heavy and take also long > time to get merged, much CPU and probably also long GC. You should check on > disk if you can observe files in the data path if some got as large as ~5g. > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 2, 2013, 11:40am UTC](https://discuss.elastic.co/t/java-io-ioexception-input-output-error-niofsindexinput/13426/6 "2013-09-02T11:40:52Z")
Under your data path you will find many files that belong to Lucene and ES internal structures, for example the .fdt files that hold the Lucene field data may give a rough estimate about how big a segment has grown. While merging, many of the Lucene index files add up because they reside in memory then. You could also try to monitor the process with pmap -x and check what files are open. Or even better, segment merge visualization tools are also available: > **[polyfractal/elasticsearch-segmentspy](https://github.com/polyfractal/elasticsearch-segmentspy)** > > elasticsearch-segmentspy - ElasticSearch plugin to watch segment dynamics (additions, merges, deletes) Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:18am UTC](https://discuss.elastic.co/t/java-io-ioexception-input-output-error-niofsindexinput/13426/7 "2017-07-06T02:18:31Z")