Question on Memory store type

Elastic Stack Elasticsearch
1

  1. If Memory store type stores indexes in the main memory, does merging
    happen in the memory?

  2. I saw some postings saying local gateway with Memory store type does not
    work. If I want to close the index with Memory store type and flush to the
    disk, which API can I call?

  3. Is there any performance data of Memory store type compared to MMap FS
    and NIO FS?

  4. I saw "outside of JVM heapspace". What does this exactly mean? If I use
    Memory store type, what is recommended JVM heap space setting? With FS
    store type, recommended setting is 50% to elasticsearch and 50% to OS disk
    cache.

Thank you!
Best, Jae

--

2

Hello Jae,

On Thu, Nov 29, 2012 at 8:53 PM, Jae metacret@gmail.com wrote:

  1. If Memory store type stores indexes in the main memory, does merging
    happen in the memory?

Yes, as far as I know.

  1. I saw some postings saying local gateway with Memory store type does
    not work.

Right. Local gateway assumes you have File System store with no memory
caching. For consistency.

If I want to close the index with Memory store type and flush to the disk,
which API can I call?

I don't think you can do that. But maybe someone else can confirm or show a
way to do it.

  1. Is there any performance data of Memory store type compared to MMap FS
    and NIO FS?

I'm not aware of such benchmarks. But I assume results would depend heavily
on lots of variables, like how often you'd index and search, how big are
your documents and indices, how much RAM you have and how fast are the
disks, etc.

  1. I saw "outside of JVM heapspace". What does this exactly mean?

The in-memory index can be either stored in JVM's memory or outside of it.
The default is to store it outside, since that should put less stress on
the GC.

If I use Memory store type, what is recommended JVM heap space setting?
With FS store type, recommended setting is 50% to elasticsearch and 50% to
OS disk cache.

It depends on the size of your index and the size of your memory. If you
store it outside the JVM heap, you might do with less than 50%. If you
store it inside, you'll probably need more. Ultimately, I think the best
way to do is to run some performance tests while monitoring your cluster -
especially in terms of memory and GC.

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

3

On Thursday, November 29, 2012 7:53:04 PM UTC+1, Jae wrote:

  1. I saw "outside of JVM heapspace". What does this exactly mean?

Note, "memory" is not necessarily equal to "heap". Elasticsearch extends
Lucene by a custom memory index type implementation. "Outside of the JVM
heap" in the docs refer to the method java.nio.ByteBuffer.allocateDirect()
in this implementation, that is, the byte buffer is stored in direct access
memory, a contiguous place for most efficient input/output by the operating
system. The direct memory is not garbage collected.

Jörg

--