Is ES es.index.store.type=memory equivalent to Lucene's RAMDirectory?


(Mark Lu) #1

I would like to use MMapDirectory at the data indexing phase (in a batch).
And then switch to index to in-memory and read only at time of serving real
user queries to optimize the search latency. I used to achieve that when
directly deal with Lucene by using RAMDirectory and read-only Searcher. Is
-Des.index.store.type=memory the ElasticSearch equivalent of ensure
RAMDirectory is used underline?

$ curl -XPUT http://localhost:9200/myindex/ -d
'
index :
store:
type: memory
'

Thanks,
-Mark

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/53b8cd87-237c-4990-80b5-25572a14a286%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #2

Yes, it is equivalent.

MMapDirectory is already using as much memory as possible, for reading data.

RamDirectory store is when you want to push all data onto the heap,
typically for volatile unit tests. For large index, it puts only burden on
the heap and your performance will suffer from GC.

Jörg

On Tue, Jun 10, 2014 at 7:43 PM, Mark Lu mark.zlu@gmail.com wrote:

I would like to use MMapDirectory at the data indexing phase (in a batch).
And then switch to index to in-memory and read only at time of serving real
user queries to optimize the search latency. I used to achieve that when
directly deal with Lucene by using RAMDirectory and read-only Searcher. Is
-Des.index.store.type=memory the ElasticSearch equivalent of ensure
RAMDirectory is used underline?

$ curl -XPUT http://localhost:9200/myindex/ -d
'
index :
store:
type: memory
'

Thanks,
-Mark

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/53b8cd87-237c-4990-80b5-25572a14a286%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/53b8cd87-237c-4990-80b5-25572a14a286%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGh8cSDzqVi5b7h7bMf621KdwC3ovWOsd-J2rHkT3DO3Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Lu) #3

Hi Jörg,

Does ES allow switching from file-based store to memory-based store without
re-indexing? We used to use file-based store to run a batch index, then
switch to read-only memory-based index when using Lucene directly. We've
found that read-only RAMDirectory is 20% faster than MmapDirecotry. That's
why we still want to use the similar type of configuration when switching
from Lucene to ES.

Thanks,
-Mark

On Tuesday, June 10, 2014 1:58:13 PM UTC-7, Jörg Prante wrote:

Yes, it is equivalent.

MMapDirectory is already using as much memory as possible, for reading
data.

RamDirectory store is when you want to push all data onto the heap,
typically for volatile unit tests. For large index, it puts only burden on
the heap and your performance will suffer from GC.

Jörg

On Tue, Jun 10, 2014 at 7:43 PM, Mark Lu <mark...@gmail.com <javascript:>>
wrote:

I would like to use MMapDirectory at the data indexing phase (in a
batch). And then switch to index to in-memory and read only at time of
serving real user queries to optimize the search latency. I used to achieve
that when directly deal with Lucene by using RAMDirectory and read-only
Searcher. Is -Des.index.store.type=memory the ElasticSearch equivalent of
ensure RAMDirectory is used underline?

$ curl -XPUT http://localhost:9200/myindex/ -d
'
index :
store:
type: memory
'

Thanks,
-Mark

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/53b8cd87-237c-4990-80b5-25572a14a286%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/53b8cd87-237c-4990-80b5-25572a14a286%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6eb553de-29b0-478f-83ae-72908275d0da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4