# File read chunk sizes **URL:** \ **Category:** Elasticsearch\ **Created:** [August 9, 2013, 7:43am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145 "2013-08-09T07:43:20Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Anand\_Nalya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anand_nalya/32/571_2.png) [@Anand\_Nalya](https://discuss.elastic.co/u/Anand_Nalya)\ **Post date:** [August 9, 2013, 7:43am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145/1 "2013-08-09T07:43:20Z")
Hi, I'm using NIOFS and I want to know what is the chunk size of the read requests given to the filesystem by ES while any search request is submitted. Is there any configuration for the same? Thanks, Anand -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 9, 2013, 9:43am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145/2 "2013-08-09T09:43:14Z")
Lucene provides a BufferedIndexInput class, where chunks are managed for reading efficiently and moved over the heap (mostly with System.arraycopy).This is a delicate task because many buffers exist while segment merging. With NIO, the buffer creation is executed in a newBuffer(byte[] buf) method which wraps the byte array into a ByteBuffer. The method setBufferSize(int newSize) of BufferedIndexInput is not exposed in Elasticsearch Store API. [http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java](http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java) Do you have issues you could solve by making the buffer size configurable in ES? Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Anand\_Nalya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anand_nalya/32/571_2.png) [@Anand\_Nalya](https://discuss.elastic.co/u/Anand_Nalya)\ **Post date:** [August 9, 2013, 9:54am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145/3 "2013-08-09T09:54:18Z")
Hi Jörg, I'm just tuning the storage for the ES. Does ES sets the buffer size to something other than 1K, if so then where? Thanks, Anand On Friday, 9 August 2013 15:13:14 UTC+5:30, Jörg Prante wrote: > Lucene provides a BufferedIndexInput class, where chunks are managed for > reading efficiently and moved over the heap (mostly with > System.arraycopy).This is a delicate task because many buffers exist while > segment merging. > > With NIO, the buffer creation is executed in a newBuffer(byte buf) > method which wraps the byte array into a ByteBuffer. > > The method setBufferSize(int newSize) of BufferedIndexInput is not > exposed in Elasticsearch Store API. > > [http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java](http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/BufferedIndexInput.java) > > Do you have issues you could solve by making the buffer size configurable > in ES? > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 9, 2013, 11:40am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145/4 "2013-08-09T11:40:20Z")
From the code in [http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/NIOFSDirectory.java](http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/NIOFSDirectory.java) and [http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java](http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java) I learned that for 64-bit JVM, the limit for setReadChunkSize() is Integer.MAX\_VALUE and can't be modified, so it is effectively ignored (at least I hope so from reading the code it's not very clear). For 32-bit JVM it is 100m by default. ES is using Lucene's directory implementations without modification. My conclusion is, to get access to the underlying BufferedIndexInput buffer size of a NIOFSDirectory, a custom NIOFSDirectory implementation would be required, with a modfied internal class NIOFSIndexInput. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Anand\_Nalya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anand_nalya/32/571_2.png) [@Anand\_Nalya](https://discuss.elastic.co/u/Anand_Nalya)\ **Post date:** [August 13, 2013, 6:40am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145/5 "2013-08-13T06:40:37Z")
Hi, I tried tracing the block read request on the disk containing the ES indices during query using blktrace. Most of the reads are of 256blocks (4KB blocks). Any idea, how is this being controlled. I'm running on 64-bit JVM. Thanks, Anand On Friday, 9 August 2013 17:10:20 UTC+5:30, Jörg Prante wrote: > From the code in > [http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/NIOFSDirectory.java](http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/NIOFSDirectory.java) > > and > [http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java](http://svn.apache.org/repos/asf/lucene/dev/trunk/lucene/core/src/java/org/apache/lucene/store/FSDirectory.java) > > I learned that for 64-bit JVM, the limit for setReadChunkSize() is > Integer.MAX\_VALUE and can't be modified, so it is effectively ignored (at > least I hope so from reading the code it's not very clear). For 32-bit JVM > it is 100m by default. > > ES is using Lucene's directory implementations without modification. > > My conclusion is, to get access to the underlying BufferedIndexInput > buffer size of a NIOFSDirectory, a custom NIOFSDirectory implementation > would be required, with a modfied internal class NIOFSIndexInput. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 13, 2013, 7:29am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145/6 "2013-08-13T07:29:36Z")
Do you have mmapfs enabled? Could be the virtual memory OS page size. If you have your file mapped into virtual memory, it can only be accessed page by page. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Anand\_Nalya](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anand_nalya/32/571_2.png) [@Anand\_Nalya](https://discuss.elastic.co/u/Anand_Nalya)\ **Post date:** [August 13, 2013, 9:05am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145/7 "2013-08-13T09:05:57Z")
mmapfs is disabled and OS page size is 4K. I've also ran the test case after clearing the page cache. got the same results. -Anand On Tuesday, 13 August 2013 12:59:36 UTC+5:30, Jörg Prante wrote: > Do you have mmapfs enabled? Could be the virtual memory OS page size. If > you have your file mapped into virtual memory, it can only be accessed page > by page. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [August 13, 2013, 10:11am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145/8 "2013-08-13T10:11:58Z")
Not sure where the 4k buffer is defined. blktrace looks at the OS calls, probably this is an optimization already implemented in the JVM. I doubt there will be much gain to increase the underlying buffer in Lucene. The buffer size is the "block size" of the data the JVM moves over to the control of the OS, and the OS has rich buffering capabilities. Increasing the JVM buffer always has the risk of double buffering. The work of [https://issues.apache.org/jira/browse/LUCENE-893](https://issues.apache.org/jira/browse/LUCENE-893) - well, it's somewhat outdated, but I think still valid - shows Lucene can keep up with even a 2k buffer. I think this post of Doug Cutting from 2004 still holds: "To my thinking, the primary role of file buffering in Lucene is to minimize the overhead of the system call itself, not to minimize physical i/o operations. Once the overhead of the system call is made insignificant, larger buffers offer little measurable improvement." [http://mail-archives.apache.org/mod\_mbox/lucene-java-user/200403.mbox/\<4069A856.70700@apache.org\>](http://mail-archives.apache.org/mod_mbox/lucene-java-user/200403.mbox/%3C4069A856.70700@apache.org%3E) Just my 2 cents. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:21am UTC](https://discuss.elastic.co/t/file-read-chunk-sizes/13145/9 "2017-07-06T02:21:23Z")
Closed on July 6, 2017, 2:21am UTC