# Pin niofs index in memory **URL:** \ **Category:** Elasticsearch\ **Created:** [March 9, 2011, 6:12pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066 "2011-03-09T18:12:47Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jacorob](https://avatars.discourse-cdn.com/v4/letter/j/dec6dc/32.png) [@jacorob](https://discuss.elastic.co/u/jacorob)\ **Post date:** [March 9, 2011, 6:12pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066/1 "2011-03-09T18:12:47Z")
I looked through the documentation for an index, but didn't see this mentioned though I easily could have missed it or not understood something. Is it possible to specify that an index should be durable (persisted to disk), but always kept in memory? For my particular use case I have a smaller index that will be heavily hit so pinning it in memory would be preferred. I can see options for storing to disk, and in-memory indexes, but not a combination of the two. Thanks, Bob ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 9, 2011, 6:28pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066/2 "2011-03-09T18:28:45Z")
You can use mmapfs: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store.html). On Wednesday, March 9, 2011 at 8:12 PM, Bob wrote: > I looked through the documentation for an index, but didn't see this > mentioned though I easily could have missed it or not understood > something. > > Is it possible to specify that an index should be durable (persisted > to disk), but always kept in memory? For my particular use case I have > a smaller index that will be heavily hit so pinning it in memory would > be preferred. I can see options for storing to disk, and in-memory > indexes, but not a combination of the two. > > Thanks, > Bob ---
**Author:** ![oreno](https://avatars.discourse-cdn.com/v4/letter/o/b4bc9f/32.png) [@oreno](https://discuss.elastic.co/u/oreno)\ **Post date:** [July 13, 2013, 12:11pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066/3 "2013-07-13T12:11:17Z")
Hi Shay, when using mmapfs, what happens to mapped addresses on ES restart? does it lose all the in memory data, and the store actually changes to a regular file system storage? or does it map all addresses once again? Thanks in advanced, Oren ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 13, 2013, 12:19pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066/4 "2013-07-13T12:19:40Z")
If you restart ES, the JVM process exits and gives all memory back to the OS, followed by another start of the JVM. There is nothing magic about mmap in this context. Jörg Am 13.07.13 14:11, schrieb oreno: > Hi Shay, > when using mmapfs, what happens to mapped addresses on ES restart? > does it lose all the in memory data, and the store actually changes to a > regular file system storage? > or does it map all addresses once again? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![oreno](https://avatars.discourse-cdn.com/v4/letter/o/b4bc9f/32.png) [@oreno](https://discuss.elastic.co/u/oreno)\ **Post date:** [July 13, 2013, 12:43pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066/5 "2013-07-13T12:43:22Z")
Thanks Jörg, I was looking for a solid in memory storage solution. I guess mmapfs is not the one. To be honest I don't really understand when it will be good practice to use explicit 'in memory' storage. It has no persistence or backup... Thanks, [image: Inline image 1] \* \* _Oren Orgad_ | Software Engineer | _eXelate [http://www.exelate.com/](http://www.exelate.com/)_ _p:_ +972- 72-2221329 | _e:_ [oreno@exelate.com](mailto:oreno@exelate.com) On Sat, Jul 13, 2013 at 3:20 PM, Jörg Prante [via Elasticsearch Users] \< [ml-node+s115913n4038073h98@n3.nabble.com](mailto:ml-node+s115913n4038073h98@n3.nabble.com)\> wrote: > If you restart ES, the JVM process exits and gives all memory back to > the OS, followed by another start of the JVM. There is nothing magic > about mmap in this context. > > Jörg > > Am 13.07.13 14:11, schrieb oreno: > > > Hi Shay, > > when using mmapfs, what happens to mapped addresses on ES restart? > > does it lose all the in memory data, and the store actually changes to a > > regular file system storage? > > or does it map all addresses once again? > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [hidden email][http://user/SendEmail.jtp?type=node&node=4038073&i=0](http://user/SendEmail.jtp?type=node&node=4038073&i=0). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > * * * > > If you reply to this email, your message will be added to the discussion > below: > > [http://elasticsearch-users.115913.n3.nabble.com/Pin-niofs-index-in-memory-tp2656438p4038073.html](http://elasticsearch-users.115913.n3.nabble.com/Pin-niofs-index-in-memory-tp2656438p4038073.html) > To unsubscribe from Pin niofs index in memory, click here[http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe\_by\_code&node=2656438&code=b3Jlbm9AZXhlbGF0ZS5jb218MjY1NjQzOHw4ODk1Mjk2Nzg=](http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2656438&code=b3Jlbm9AZXhlbGF0ZS5jb218MjY1NjQzOHw4ODk1Mjk2Nzg=) > . > NAML[http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro\_viewer&id=instant\_html!nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify\_subscribers!nabble%3Aemail.naml-instant\_emails!nabble%3Aemail.naml-send\_instant\_email!nabble%3Aemail.naml](http://elasticsearch-users.115913.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml) ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 13, 2013, 4:34pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066/6 "2013-07-13T16:34:24Z")
Search methods in ES are "in memory" by default, regardless to mmapfs, niofs, simplefs etc. mmapfs uses OS memory optimizations for file reading in addition to niofs, that is pretty all about it. The "store" concept [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store/) is related to writing data only. There is a "memory" (RAM only) store option, for keeping volatile data in the JVM heap. This is useful in the case you only test ES (with junit/testng, you don't create temp files) or for data from a persistent source, where recreating by bulk indexing is faster/easier than gateway storage (which is rarely the case). The "memory" option must not be confused with search caches for filter/facets which are much more important for performance. If you mount a filesystem on a RAM disk and use mmapfs/niofs by pointing the "path.data" config to it, you can pretty much achieve a better effect than "memory" store, because OS might optimize I/O and Java might avoid moving data over the JVM heap. Jörg Am 13.07.13 14:43, schrieb oreno: > Thanks Jörg, > I was looking for a solid in memory storage solution. I guess mmapfs > is not the one. > To be honest I don't really understand when it will be good practice > to use explicit 'in memory' storage. > It has no persistence or backup... -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![oreno](https://avatars.discourse-cdn.com/v4/letter/o/b4bc9f/32.png) [@oreno](https://discuss.elastic.co/u/oreno)\ **Post date:** [July 13, 2013, 4:57pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066/7 "2013-07-13T16:57:18Z")
Thanks for the input. 1.I actually not looking for cache solution since my queries are pretty dynamic. 2.Won't mounting to RAM results in the lose of data once the ES restarts, same as in memory method? 3.Are you saying that mmapfs as no affect on search performance? I only tried it because I read that it has in memory mapping along with file system storage that I require for persistence. 4.What would you say is the best setting to have when you need the best search performance along with persistence for the indexed data? Thanks, Sent from my iPad On 13.07.2013, at 19:35, Jörg Prante [via Elasticsearch Users] [ml-node+s115913n4038080h15@n3.nabble.com](mailto:ml-node+s115913n4038080h15@n3.nabble.com) wrote: > Search methods in ES are "in memory" by default, regardless to mmapfs, > niofs, simplefs etc. > > mmapfs uses OS memory optimizations for file reading in addition to > niofs, that is pretty all about it. > > The "store" concept > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/store/) is > related to writing data only. > > There is a "memory" (RAM only) store option, for keeping volatile data > in the JVM heap. This is useful in the case you only test ES (with > junit/testng, you don't create temp files) or for data from a persistent > source, where recreating by bulk indexing is faster/easier than gateway > storage (which is rarely the case). The "memory" option must not be > confused with search caches for filter/facets which are much more > important for performance. > > If you mount a filesystem on a RAM disk and use mmapfs/niofs by pointing > the "path.data" config to it, you can pretty much achieve a better > effect than "memory" store, because OS might optimize I/O and Java might > avoid moving data over the JVM heap. > > Jörg > > Am 13.07.13 14:43, schrieb oreno: > > > Thanks Jörg, > > I was looking for a solid in memory storage solution. I guess mmapfs > > is not the one. > > To be honest I don't really understand when it will be good practice > > to use explicit 'in memory' storage. > > It has no persistence or backup... > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > If you reply to this email, your message will be added to the discussion below: > [http://elasticsearch-users.115913.n3.nabble.com/Pin-niofs-index-in-memory-tp2656438p4038080.html](http://elasticsearch-users.115913.n3.nabble.com/Pin-niofs-index-in-memory-tp2656438p4038080.html) > To unsubscribe from Pin niofs index in memory, click here. > NAML ---
**Author:** ![unknownunknown](https://avatars.discourse-cdn.com/v4/letter/u/4da419/32.png) [@unknownunknown](https://discuss.elastic.co/u/unknownunknown)\ **Post date:** [August 19, 2016, 6:38pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066/8 "2016-08-19T18:38:08Z")
> [@oreno](#): > > 2.Won't mounting to RAM results in the lose of data once the ES restarts, same as in memory method? A ramfs will persist until the kernel reboots or the filesystem is otherwise unmounted. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:26pm UTC](https://discuss.elastic.co/t/pin-niofs-index-in-memory/4066/9 "2017-07-05T22:26:34Z")