I'm using NIOFS and I want to know what is the chunk size of the read
requests given to the filesystem by ES while any search request is
submitted. Is there any configuration for the same?
Lucene provides a BufferedIndexInput class, where chunks are managed for
reading efficiently and moved over the heap (mostly with
System.arraycopy).This is a delicate task because many buffers exist while
segment merging.
With NIO, the buffer creation is executed in a newBuffer(byte[] buf) method
which wraps the byte array into a ByteBuffer.
The method setBufferSize(int newSize) of BufferedIndexInput is not exposed
in Elasticsearch Store API.
I'm just tuning the storage for the ES. Does ES sets the buffer size to
something other than 1K, if so then where?
Thanks,
Anand
On Friday, 9 August 2013 15:13:14 UTC+5:30, Jörg Prante wrote:
Lucene provides a BufferedIndexInput class, where chunks are managed for
reading efficiently and moved over the heap (mostly with
System.arraycopy).This is a delicate task because many buffers exist while
segment merging.
With NIO, the buffer creation is executed in a newBuffer(byte buf)
method which wraps the byte array into a ByteBuffer.
The method setBufferSize(int newSize) of BufferedIndexInput is not
exposed in Elasticsearch Store API.
I learned that for 64-bit JVM, the limit for setReadChunkSize() is
Integer.MAX_VALUE and can't be modified, so it is effectively ignored (at
least I hope so from reading the code it's not very clear). For 32-bit JVM
it is 100m by default.
ES is using Lucene's directory implementations without modification.
My conclusion is, to get access to the underlying BufferedIndexInput buffer
size of a NIOFSDirectory, a custom NIOFSDirectory implementation would be
required, with a modfied internal class NIOFSIndexInput.
I tried tracing the block read request on the disk containing the ES
indices during query using blktrace. Most of the reads are of 256blocks
(4KB blocks). Any idea, how is this being controlled. I'm running on 64-bit
JVM.
Thanks,
Anand
On Friday, 9 August 2013 17:10:20 UTC+5:30, Jörg Prante wrote:
I learned that for 64-bit JVM, the limit for setReadChunkSize() is
Integer.MAX_VALUE and can't be modified, so it is effectively ignored (at
least I hope so from reading the code it's not very clear). For 32-bit JVM
it is 100m by default.
ES is using Lucene's directory implementations without modification.
My conclusion is, to get access to the underlying BufferedIndexInput
buffer size of a NIOFSDirectory, a custom NIOFSDirectory implementation
would be required, with a modfied internal class NIOFSIndexInput.
Do you have mmapfs enabled? Could be the virtual memory OS page size. If
you have your file mapped into virtual memory, it can only be accessed page
by page.
mmapfs is disabled and OS page size is 4K. I've also ran the test case
after clearing the page cache. got the same results.
-Anand
On Tuesday, 13 August 2013 12:59:36 UTC+5:30, Jörg Prante wrote:
Do you have mmapfs enabled? Could be the virtual memory OS page size. If
you have your file mapped into virtual memory, it can only be accessed page
by page.
Not sure where the 4k buffer is defined. blktrace looks at the OS calls,
probably this is an optimization already implemented in the JVM.
I doubt there will be much gain to increase the underlying buffer in
Lucene. The buffer size is the "block size" of the data the JVM moves over
to the control of the OS, and the OS has rich buffering capabilities.
Increasing the JVM buffer always has the risk of double buffering.
I think this post of Doug Cutting from 2004 still holds:
"To my thinking, the primary role of file buffering in Lucene is to
minimize the overhead of the system call itself, not to minimize physical
i/o operations. Once the overhead of the system call is made
insignificant, larger buffers offer little measurable improvement."
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.