Pin niofs index in memory

Elastic Stack Elasticsearch
1

I looked through the documentation for an index, but didn't see this
mentioned though I easily could have missed it or not understood
something.

Is it possible to specify that an index should be durable (persisted
to disk), but always kept in memory? For my particular use case I have
a smaller index that will be heavily hit so pinning it in memory would
be preferred. I can see options for storing to disk, and in-memory
indexes, but not a combination of the two.

Thanks,
Bob

2

You can use mmapfs: Elasticsearch Platform — Find real-time answers at scale | Elastic.
On Wednesday, March 9, 2011 at 8:12 PM, Bob wrote:

I looked through the documentation for an index, but didn't see this
mentioned though I easily could have missed it or not understood
something.

Is it possible to specify that an index should be durable (persisted
to disk), but always kept in memory? For my particular use case I have
a smaller index that will be heavily hit so pinning it in memory would
be preferred. I can see options for storing to disk, and in-memory
indexes, but not a combination of the two.

Thanks,
Bob

3

Hi Shay,
when using mmapfs, what happens to mapped addresses on ES restart?
does it lose all the in memory data, and the store actually changes to a regular file system storage?
or does it map all addresses once again?

Thanks in advanced,

Oren

4

If you restart ES, the JVM process exits and gives all memory back to
the OS, followed by another start of the JVM. There is nothing magic
about mmap in this context.

Jörg

Am 13.07.13 14:11, schrieb oreno:

Hi Shay,
when using mmapfs, what happens to mapped addresses on ES restart?
does it lose all the in memory data, and the store actually changes to a
regular file system storage?
or does it map all addresses once again?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Thanks Jörg,
I was looking for a solid in memory storage solution. I guess mmapfs is not
the one.
To be honest I don't really understand when it will be good practice to use
explicit 'in memory' storage.
It has no persistence or backup...

Thanks,

[image: Inline image 1]
*
*
Oren Orgad | Software Engineer | eXelate http://www.exelate.com/

p: +972- 72-2221329 | e: oreno@exelate.com

On Sat, Jul 13, 2013 at 3:20 PM, Jörg Prante [via Elasticsearch Users] <
ml-node+s115913n4038073h98@n3.nabble.com> wrote:

If you restart ES, the JVM process exits and gives all memory back to
the OS, followed by another start of the JVM. There is nothing magic
about mmap in this context.

Jörg

Am 13.07.13 14:11, schrieb oreno:

Hi Shay,
when using mmapfs, what happens to mapped addresses on ES restart?
does it lose all the in memory data, and the store actually changes to a
regular file system storage?
or does it map all addresses once again?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to [hidden email]http://user/SendEmail.jtp?type=node&node=4038073&i=0.

For more options, visit https://groups.google.com/groups/opt_out.

If you reply to this email, your message will be added to the discussion
below:

http://elasticsearch-users.115913.n3.nabble.com/Pin-niofs-index-in-memory-tp2656438p4038073.html
To unsubscribe from Pin niofs index in memory, click herehttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2656438&code=b3Jlbm9AZXhlbGF0ZS5jb218MjY1NjQzOHw4ODk1Mjk2Nzg=
.
NAMLhttp://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers!nabble%3Aemail.naml-instant_emails!nabble%3Aemail.naml-send_instant_email!nabble%3Aemail.naml

6

Search methods in ES are "in memory" by default, regardless to mmapfs,
niofs, simplefs etc.

mmapfs uses OS memory optimizations for file reading in addition to
niofs, that is pretty all about it.

The "store" concept
Elasticsearch Platform — Find real-time answers at scale | Elastic is
related to writing data only.

There is a "memory" (RAM only) store option, for keeping volatile data
in the JVM heap. This is useful in the case you only test ES (with
junit/testng, you don't create temp files) or for data from a persistent
source, where recreating by bulk indexing is faster/easier than gateway
storage (which is rarely the case). The "memory" option must not be
confused with search caches for filter/facets which are much more
important for performance.

If you mount a filesystem on a RAM disk and use mmapfs/niofs by pointing
the "path.data" config to it, you can pretty much achieve a better
effect than "memory" store, because OS might optimize I/O and Java might
avoid moving data over the JVM heap.

Jörg

Am 13.07.13 14:43, schrieb oreno:

Thanks Jörg,
I was looking for a solid in memory storage solution. I guess mmapfs
is not the one.
To be honest I don't really understand when it will be good practice
to use explicit 'in memory' storage.
It has no persistence or backup...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Thanks for the input.
1.I actually not looking for cache solution since my queries are pretty dynamic.
2.Won't mounting to RAM results in the lose of data once the ES restarts, same as in memory method?

3.Are you saying that mmapfs as no affect on search performance?
I only tried it because I read that it has in memory mapping along with file system storage
that I require for persistence.

4.What would you say is the best setting to have when you need the best search performance along with persistence for the indexed data?

Thanks,
Sent from my iPad

On 13.07.2013, at 19:35, Jörg Prante [via Elasticsearch Users] ml-node+s115913n4038080h15@n3.nabble.com wrote:

Search methods in ES are "in memory" by default, regardless to mmapfs,
niofs, simplefs etc.

mmapfs uses OS memory optimizations for file reading in addition to
niofs, that is pretty all about it.

The "store" concept
Elasticsearch Platform — Find real-time answers at scale | Elastic is
related to writing data only.

There is a "memory" (RAM only) store option, for keeping volatile data
in the JVM heap. This is useful in the case you only test ES (with
junit/testng, you don't create temp files) or for data from a persistent
source, where recreating by bulk indexing is faster/easier than gateway
storage (which is rarely the case). The "memory" option must not be
confused with search caches for filter/facets which are much more
important for performance.

If you mount a filesystem on a RAM disk and use mmapfs/niofs by pointing
the "path.data" config to it, you can pretty much achieve a better
effect than "memory" store, because OS might optimize I/O and Java might
avoid moving data over the JVM heap.

Jörg

Am 13.07.13 14:43, schrieb oreno:

Thanks Jörg,
I was looking for a solid in memory storage solution. I guess mmapfs
is not the one.
To be honest I don't really understand when it will be good practice
to use explicit 'in memory' storage.
It has no persistence or backup...

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/groups/opt_out.

If you reply to this email, your message will be added to the discussion below:
http://elasticsearch-users.115913.n3.nabble.com/Pin-niofs-index-in-memory-tp2656438p4038080.html
To unsubscribe from Pin niofs index in memory, click here.
NAML

8

A ramfs will persist until the kernel reboots or the filesystem is otherwise unmounted.