Benchmarks: default (niofs) vs memory mapped (mmapfs)

Elastic Stack Elasticsearch
1

Anybody have any benchmarks for the default niofs vs the memory mapped
mmapfs? Haven't seen any. One or two StackOverflow posts said no
difference.
I'm starting to run some benchmarks myself and initial results suggest no
significant difference when there's plenty of memory.

Here's my setup:

client --> Rails app --> single elasticsearch node (0.19.12)

The elasticsearch node is running on a memory optimized ec2 instance
(m2.2xlarge, 34 GB RAM).

I have two indexes, one default, one mmapfs. Otherwise identical. I'm
indexing them both as we speak, around 300k documents right now.

Running my client, I see maybe a couple of millisecond difference in "took"
time (the time elasticsearch reports in the response). The total response
time (taking into account the Rails app, the HTTP transfer back to the
client) shows a larger difference. But the total response time has too many
variables so I'm ignoring it for now until I can isolate things better.

The index is much smaller than available memory (at most probably 2 GB
each). The elasticsearch node is running w/ 8 GB heap, mlockall true.
I'm guessing since there's plenty of available memory, the minor diff shows
the OS file cache is very good. Maybe I need to run it when the index size
is greater than physical memory, to see how paging and file caches behave.
Maybe then the difference will be more significant.

Anyways, if anybody has some benchmarks or data, would love to heard.

Took time is the first number:

#0 simple, default_mmap, false , 16, 92, 35169
#0 simple_full_doc, default_mmap, false , 15, 321, 35537
#0 filter, default_mmap, false , 17, 99, 0
#0 filter_full_doc, default_mmap, false , 16, 109, 0
#0 facets, default_mmap, false , 31, 129, 36716
#0 facets_full_doc, default_mmap, false , 30, 536, 37103
#0 filter_facets, default_mmap, false , 26, 154, 0
#0 gen_filters_facets_full_doc, default_mmap, false , 32, 108, 0

#0 simple, default, false , 18, 86, 35170
#0 simple_full_doc, default, false , 16, 350, 35540
#0 filter, default, false , 20, 103, 0
#0 filter_full_doc, default, false , 20, 109, 0
#0 facets, default, false , 35, 178, 36720
#0 facets_full_doc, default, false , 34, 697, 37102
#0 filter_facets, default, false , 27, 161, 0
#0 gen_filters_facets_full_doc, default, false , 34, 114, 0

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Correct. The Lucene index file sizes should exceed available RAM by far and
indexing should be very active to generate a lot of new segment files to
see any effect of mmapfs.

Jörg

On Thu, Aug 8, 2013 at 7:51 AM, Tinou Bao tinou.bao@gmail.com wrote:

Maybe I need to run it when the index size is greater than physical
memory, to see how paging and file caches behave.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks Jörg for the input. Here's an interesting curve ball. When the index
was ~300K docs (and growing), mmapfs was faster across the board (query,
filter, facet). Now that my index is at ~700k docs (and growing), query and
filter are still faster in mmapfs. But, searches w/ facets are faster w/
the default niofs. Very consistent that mmapfs is slower when faceting is
involved. Need to investigate further.

On Thursday, August 8, 2013 1:42:50 AM UTC-7, Jörg Prante wrote:

Correct. The Lucene index file sizes should exceed available RAM by far
and indexing should be very active to generate a lot of new segment files
to see any effect of mmapfs.

Jörg

On Thu, Aug 8, 2013 at 7:51 AM, Tinou Bao <tino...@gmail.com <javascript:>

wrote:

Maybe I need to run it when the index size is greater than physical
memory, to see how paging and file caches behave.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Just to close the loop...looks like the ec2 instance wasn't configured
properly with respect to disk space and swap space. Re-ran my tests against
my current non-ec2 load environment and mmapfs does not perform worst for
faceting.

-T

On Thursday, August 8, 2013 9:33:34 AM UTC-7, Tinou Bao wrote:

Thanks Jörg for the input. Here's an interesting curve ball. When the
index was ~300K docs (and growing), mmapfs was faster across the board
(query, filter, facet). Now that my index is at ~700k docs (and growing),
query and filter are still faster in mmapfs. But, searches w/ facets are
faster w/ the default niofs. Very consistent that mmapfs is slower when
faceting is involved. Need to investigate further.

On Thursday, August 8, 2013 1:42:50 AM UTC-7, Jörg Prante wrote:

Correct. The Lucene index file sizes should exceed available RAM by far
and indexing should be very active to generate a lot of new segment files
to see any effect of mmapfs.

Jörg

On Thu, Aug 8, 2013 at 7:51 AM, Tinou Bao tino...@gmail.com wrote:

Maybe I need to run it when the index size is greater than physical
memory, to see how paging and file caches behave.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.