# In-memory indices **URL:** \ **Category:** Elasticsearch\ **Created:** [September 16, 2011, 1:41am UTC](https://discuss.elastic.co/t/in-memory-indices/5391 "2011-09-16T01:41:34Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![rcch](https://avatars.discourse-cdn.com/v4/letter/r/4af34b/32.png) [@rcch](https://discuss.elastic.co/u/rcch)\ **Post date:** [September 16, 2011, 1:41am UTC](https://discuss.elastic.co/t/in-memory-indices/5391/1 "2011-09-16T01:41:34Z")
Hi all, 1. I set up an index with -Des.index.storage.type=memory. 2. I added a bunch of documents. 3. Then, I killed elastic search 4. I brought it back up and queried the index - and the searches were sucessful. I believe there's a gap in my understanding. If the index is an in- memory index - I thought there is no persistence in this case. But seems like there is a built in layer of persistence. I use the default configuration (did not set up a gateway, etc). So the questions are: 1. Is there persistence for in-memory indices ? If not, how did my example work ? 2. What happens if I go over the index limit for an in-memory index ? Will ES gracefully transition between keeping things in RAM and going to diske or will it die ? 3. How do I set up a hybrid where I keep a considerable amount of stuff in RAM (cached) and go to diske when needed? Thank you for your help. Cheers, Vijay ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [September 16, 2011, 8:36am UTC](https://discuss.elastic.co/t/in-memory-indices/5391/2 "2011-09-16T08:36:13Z")
in-memory means that ES will nevertheless makes backup (via local gateway) of the data to recover from that in a case of an incident. But if the data does not fit completely into memory ES will throw at some point OutOfMem errors. No 'transition' is done. What you probably want is the default disc base index, which is also in-memory for parts of the data (ES does a lot caching but also the Operating system) and so it'll require less RAM and reads from disc if the data is not cached Regards, Peter. On 16 Sep., 03:41, rcch [vijay....@gmail.com](mailto:vijay....@gmail.com) wrote: > Hi all, > > 1. I set up an index with -Des.index.storage.type=memory. > 2. I added a bunch of documents. > 3. Then, I killed Elasticsearch > 4. I brought it back up and queried the index - and the searches were > sucessful. > > I believe there's a gap in my understanding. If the index is an in- > memory index - I thought there is no persistence in this case. But > seems like there is a built in layer of persistence. I use the default > configuration (did not set up a gateway, etc). > > So the questions are: > > 1. Is there persistence for in-memory indices ? If not, how did my > example work ? > > 2. What happens if I go over the index limit for an in-memory index ? > Will ES gracefully transition between keeping things in RAM and going > to diske or will it die ? > > 3. How do I set up a hybrid where I keep a considerable amount of > stuff in RAM (cached) and go to diske when needed? > > Thank you for your help. > > Cheers, > > Vijay ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 16, 2011, 6:52pm UTC](https://discuss.elastic.co/t/in-memory-indices/5391/3 "2011-09-16T18:52:40Z")
There is no persistency with in memory indices and local gateway, only with shared gateway. Are you sure you got results? Also, did you do a full cluster shutdown (if you are running more than one node)? Note, there are bugs in the in memory (outside of JVM heap) store which I did not manage yet to track down. Use the file system ones, or mmapfs, it should be fast enough. If not, you can set the store to "ram" which defaults to Lucene in memory (in heap) memory store. On Fri, Sep 16, 2011 at 11:36 AM, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)wrote: > in-memory means that ES will nevertheless makes backup (via local > gateway) of the data to recover from that in a case of an incident. > But if the data does not fit completely into memory ES will throw at > some point OutOfMem errors. No 'transition' is done. > > What you probably want is the default disc base index, which is also > in-memory for parts of the data (ES does a lot caching but also the > Operating system) and so it'll require less RAM and reads from disc if > the data is not cached > > Regards, > Peter. > > On 16 Sep., 03:41, rcch [vijay....@gmail.com](mailto:vijay....@gmail.com) wrote: > > > Hi all, > > > > 1. I set up an index with -Des.index.storage.type=memory. > > 2. I added a bunch of documents. > > 3. Then, I killed Elasticsearch > > 4. I brought it back up and queried the index - and the searches were > > sucessful. > > > > I believe there's a gap in my understanding. If the index is an in- > > memory index - I thought there is no persistence in this case. But > > seems like there is a built in layer of persistence. I use the default > > configuration (did not set up a gateway, etc). > > > > So the questions are: > > > > 1. Is there persistence for in-memory indices ? If not, how did my > > example work ? > > > > 2. What happens if I go over the index limit for an in-memory index ? > > Will ES gracefully transition between keeping things in RAM and going > > to diske or will it die ? > > > > 3. How do I set up a hybrid where I keep a considerable amount of > > stuff in RAM (cached) and go to diske when needed? > > > > Thank you for your help. > > > > Cheers, > > > > Vijay ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [September 18, 2011, 12:08pm UTC](https://discuss.elastic.co/t/in-memory-indices/5391/4 "2011-09-18T12:08:43Z")
> There is no persistency with in memory indices and local gateway, only with > shared gateway. After some digging deeper now I think I have a better understanding. So @Vijay sorry for the confusion. @shay is the following assumption/ explanation correct? When using local gateway the index is simply stored on disc and when restarting the node it will use this information and the transaction log to recover the index, right? If the index is stored in-memory then there is no data to recover from. When using shared gateway there is an additional storage (periodically writing to this storage?). Then the node can simply recover the in- memory index from that storage. Now how does the transaction log comes into the game here? Does it exist for in-memory indices at all? Regards, Peter. ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [September 18, 2011, 12:15pm UTC](https://discuss.elastic.co/t/in-memory-indices/5391/5 "2011-09-18T12:15:13Z")
One further question: to force Elasticsearch to recover from the local disc for an in-memory index then I can simply use the shared gateway with the default _local_ work directory, right? I'm a bit confused of the gateway naming (local vs. shared ...) On 18 Sep., 14:08, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com) wrote: > > There is no persistency with in memory indices and local gateway, only with > > shared gateway. > > After some digging deeper now I think I have a better understanding. > So @Vijay sorry for the confusion. @shay is the following assumption/ > explanation correct? > > When using local gateway the index is simply stored on disc and when > restarting the node it will use this information and the transaction > log to recover the index, right? If the index is stored in-memory then > there is no data to recover from. > > When using shared gateway there is an additional storage (periodically > writing to this storage?). Then the node can simply recover the in- > memory index from that storage. > > Now how does the transaction log comes into the game here? Does it > exist for in-memory indices at all? > > Regards, > Peter. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 18, 2011, 12:58pm UTC](https://discuss.elastic.co/t/in-memory-indices/5391/6 "2011-09-18T12:58:23Z")
Yes, when using a shared gateway, periodically, the state of the indices are persisted to the shared storage location. Transaction log is still in play when using in memory indices since they play important part not just in making sure indexed data does not get lost, but also when doing both shared gateway persistency (snapshot), and when doing peer shard recovery. On Sun, Sep 18, 2011 at 3:08 PM, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)wrote: > > There is no persistency with in memory indices and local gateway, only > > with > > shared gateway. > > After some digging deeper now I think I have a better understanding. > So @Vijay sorry for the confusion. @shay is the following assumption/ > explanation correct? > > When using local gateway the index is simply stored on disc and when > restarting the node it will use this information and the transaction > log to recover the index, right? If the index is stored in-memory then > there is no data to recover from. > > When using shared gateway there is an additional storage (periodically > writing to this storage?). Then the node can simply recover the in- > memory index from that storage. > > Now how does the transaction log comes into the game here? Does it > exist for in-memory indices at all? > > Regards, > Peter. ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [September 18, 2011, 3:54pm UTC](https://discuss.elastic.co/t/in-memory-indices/5391/7 "2011-09-18T15:54:20Z")
Thanks for the explantion and confirmation. > but also when doing both shared gateway persistency (snapshot) But when doing local gateway (for an in memory index) there is no snapshot thing, right? > and when doing peer shard recovery What do you mean here? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [September 18, 2011, 4:27pm UTC](https://discuss.elastic.co/t/in-memory-indices/5391/8 "2011-09-18T16:27:49Z")
On Sun, Sep 18, 2011 at 6:54 PM, Karussell [tableyourtime@googlemail.com](mailto:tableyourtime@googlemail.com)wrote: > Thanks for the explantion and confirmation. > > > but also when doing both shared gateway persistency (snapshot) > > But when doing local gateway (for an in memory index) there is no > snapshot thing, right? Right, local gateway does not need to snapshot since it can recover from the actual state of the indices. > > and when doing peer shard recovery > > What do you mean here? When you increase the shards replicas, or shard migrate from one node to another, they do recovery from one another. Thats what I mean by peer recovery. ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [September 18, 2011, 7:45pm UTC](https://discuss.elastic.co/t/in-memory-indices/5391/9 "2011-09-18T19:45:14Z")
Thanks! makes sense! On 18 Sep., 18:27, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > On Sun, Sep 18, 2011 at 6:54 PM, Karussell [tableyourt...@googlemail.com](mailto:tableyourt...@googlemail.com)wrote: > > > Thanks for the explantion and confirmation. > > > > but also when doing both shared gateway persistency (snapshot) > > > But when doing local gateway (for an in memory index) there is no > > snapshot thing, right? > > Right, local gateway does not need to snapshot since it can recover from the > actual state of the indices. > > > > and when doing peer shard recovery > > > What do you mean here? > > When you increase the shards replicas, or shard migrate from one node to > another, they do recovery from one another. Thats what I mean by peer > recovery. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:54am UTC](https://discuss.elastic.co/t/in-memory-indices/5391/10 "2017-07-06T03:54:12Z")