Java.io.IOException: Input/output error: NIOFSIndexInput

Elastic Stack Elasticsearch
1

Hello, recently we receive a lot of these error message in our
elasticsearch nodes.Initially, we think that might be due to hardware issue
and we replace the disk but we are still seeing this issue
now and then. It occur more frequently lately. Should we worry on this
exception? What actually is happening and how can we prevent this from
happening?

A little background of our environment, using elasticsearch version 0.20.5
and 9 gc nodes in total.

[2013-08-30 15:49:45,794][WARN ][index.merge.scheduler ] [node1]
[MyIndex][57] failed to merge
java.io.IOException: Input/output error:
NIOFSIndexInput(path="/vol/elasticsearch/MyIndex_Cluster/nodes/0/indices/MyIndex/57/index/_12sww.fdx")
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:180)
at
org.apache.lucene.store.BufferedIndexInput.refill(BufferedIndexInput.java:270)
at
org.apache.lucene.store.BufferedIndexInput.readByte(BufferedIndexInput.java:40)
at org.apache.lucene.store.DataInput.readInt(DataInput.java:86)
at
org.apache.lucene.store.BufferedIndexInput.readInt(BufferedIndexInput.java:179)
at org.apache.lucene.store.DataInput.readLong(DataInput.java:130)
at
org.apache.lucene.store.BufferedIndexInput.readLong(BufferedIndexInput.java:192)
at
org.apache.lucene.index.FieldsReader.rawDocs(FieldsReader.java:292)
at
org.apache.lucene.index.SegmentMerger.copyFieldsWithDeletions(SegmentMerger.java:265)
at
org.apache.lucene.index.SegmentMerger.mergeFields(SegmentMerger.java:223)
at
org.apache.lucene.index.SegmentMerger.merge(SegmentMerger.java:107)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4263)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3908)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:388)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:91)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:456)
Caused by: java.io.IOException: Input/output error
at sun.nio.ch.FileDispatcher.pread0(Native Method)
at sun.nio.ch.FileDispatcher.pread(FileDispatcher.java:31)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:199)
at sun.nio.ch.IOUtil.read(IOUtil.java:175)
at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:612)
at
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.readInternal(NIOFSDirectory.java:162)
... 15 more

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

You have a RAM issue. Input of data fails because your RAM for I/O is too
small, while segment merging is going on.

Consider decreasing the maximum segment size for merging (default is 5g, my
recommendation is 2g)

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi Jörg,

Thank you for your insight. When you said it is RAM issue, I take it as the
amount of heap allocated to elasticsearch jvm? Read into merging
documentation here,
Elasticsearch Platform — Find real-time answers at scale | Elastic , we are
currently using the default 5g, will increase the jvm heap twice (from 8G
to 16G) prevent this issue from happening again?

Jason

On Monday, September 2, 2013 3:34:41 PM UTC+8, Jörg Prante wrote:

You have a RAM issue. Input of data fails because your RAM for I/O is too
small, while segment merging is going on.

Consider decreasing the maximum segment size for merging (default is 5g,
my recommendation is 2g)

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

If you have enough RAM outside JVM available to increase the heap space, it
may help. It depends on your requirements for processing large segment
merges. In my view, 5g sized segments are a bit heavy and take also long
time to get merged, much CPU and probably also long GC. You should check on
disk if you can observe files in the data path if some got as large as ~5g.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi Jörg,

Yes, in the meantime, we have sufficient memory to a single node which each
has 32GB memory in total. Notice that the offheap memory usage always hover
around 40-60MB.. so the merging should have been done in the jvm heap. Any
specific files in in
/vol/elasticsearch/MyIndex_Cluster/nodes/0/indices/MyIndex/[dd]
should we monitor in particular or as a collective summation?

Jason

On Mon, Sep 2, 2013 at 5:43 PM, joergprante@gmail.com <joergprante@gmail.com

wrote:

If you have enough RAM outside JVM available to increase the heap space,
it may help. It depends on your requirements for processing large segment
merges. In my view, 5g sized segments are a bit heavy and take also long
time to get merged, much CPU and probably also long GC. You should check on
disk if you can observe files in the data path if some got as large as ~5g.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Under your data path you will find many files that belong to Lucene and ES
internal structures, for example the .fdt files that hold the Lucene field
data may give a rough estimate about how big a segment has grown. While
merging, many of the Lucene index files add up because they reside in
memory then. You could also try to monitor the process with pmap -x and
check what files are open. Or even better, segment merge visualization
tools are also available:

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.