# How to move the whole data to main memory
**URL:** https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766
**Category:** Elasticsearch
**Created:** [August 17, 2012, 9:38am UTC](https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766 "2012-08-17T09:38:57Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Jalal](https://discuss.elastic.co/u/Jalal)
#### Post date: [August 17, 2012, 9:38am UTC](https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766/1 "2012-08-17T09:38:57Z")
Hi,
I am trying to leverage max performance out of elasticsearch by moving the
whole data to main memory, my whole data set and index are like 2GB and I
have huge amount of ram, I want to move everything to main memory.
My current setting is index.storage.type as memory and used
gateway.type=fs.
While indexing everything is stored in main memory, but when I restart
elasticsearch, it doesn't move the index to main memory(RAM).
I have the following settings in my elasticsearch.yml
cluster.name: elasticsearchtest
index.storage.type: memory
cache.memory.small\_buffer\_size: 1mb
cache.memory.large\_buffer\_size: 10mb
cache.memory.small\_cache\_size: 1000mb
cache.memory.large\_cache\_size: 2000mb
gateway.type: fs
gateway.fs.location: /home/algotree/deltaES
--
---
### Author:  [@otisg](https://discuss.elastic.co/u/otisg)
#### Post date: [August 20, 2012, 10:57pm UTC](https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766/2 "2012-08-20T22:57:55Z")
Hi,
If index.storage.type: memory doesn't do it, you could always do something
like query for _:_ and force the index to be cached by the OS.
## Otis
Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html)
Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html)
On Friday, August 17, 2012 5:38:57 AM UTC-4, Jalal wrote:
> Hi,
> I am trying to leverage max performance out of elasticsearch by moving
> the whole data to main memory, my whole data set and index are like 2GB and
> I have huge amount of ram, I want to move everything to main memory.
>
> My current setting is index.storage.type as memory and used
> gateway.type=fs.
>
> While indexing everything is stored in main memory, but when I restart
> elasticsearch, it doesn't move the index to main memory(RAM).
>
> I have the following settings in my elasticsearch.yml
>
> cluster.name: elasticsearchtest
> index.storage.type: memory
> cache.memory.small\_buffer\_size: 1mb
> cache.memory.large\_buffer\_size: 10mb
> cache.memory.small\_cache\_size: 1000mb
> cache.memory.large\_cache\_size: 2000mb
> gateway.type: fs
> gateway.fs.location: /home/algotree/deltaES
--
---
### Author:  [@Jalal](https://discuss.elastic.co/u/Jalal)
#### Post date: [August 31, 2012, 10:04am UTC](https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766/3 "2012-08-31T10:04:28Z")
On Tuesday, August 21, 2012 4:27:55 AM UTC+5:30, Otis Gospodnetic wrote:
> Hi,
>
> If index.storage.type: memory doesn't do it, you could always do something
> like query for _:_ and force the index to be cached by the OS.
>
> ## Otis
>
> Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html)
> Scalable Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html)
>
> On Friday, August 17, 2012 5:38:57 AM UTC-4, Jalal wrote:
>
> > Hi,
> > I am trying to leverage max performance out of elasticsearch by moving
> > the whole data to main memory, my whole data set and index are like 2GB and
> > I have huge amount of ram, I want to move everything to main memory.
> >
> > My current setting is index.storage.type as memory and used
> > gateway.type=fs.
> >
> > While indexing everything is stored in main memory, but when I restart
> > elasticsearch, it doesn't move the index to main memory(RAM).
> >
> > I have the following settings in my elasticsearch.yml
> >
> > cluster.name: elasticsearchtest
> > index.storage.type: memory
> > cache.memory.small\_buffer\_size: 1mb
> > cache.memory.large\_buffer\_size: 10mb
> > cache.memory.small\_cache\_size: 1000mb
> > cache.memory.large\_cache\_size: 2000mb
> > gateway.type: fs
> > gateway.fs.location: /home/algotree/deltaES
Thanks Otis,
I was able to store the entire index to RAM by using the setting below
while indexing the data but now the problem is the RAM usage by
Elasticsearch is almost three times the size of the index.
store.type" : "memory"
gateway.type: fs
--
---
### Author:  [@Ivan](https://discuss.elastic.co/u/Ivan)
#### Post date: [September 4, 2012, 2:39am UTC](https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766/4 "2012-09-04T02:39:40Z")
Lucene will often use up to three times the size of the index due to
the merging of existing segments. Segments are merged in the
background in paralel with the existing index, so more disk (if using
a disk based store) or memory will be used. How many updates are you
doing?
--
Ivan
On Fri, Aug 31, 2012 at 3:04 AM, Jalal [JalalM@algotree.com](mailto:JalalM@algotree.com) wrote:
> Thanks Otis,
>
> I was able to store the entire index to RAM by using the setting below while
> indexing the data but now the problem is the RAM usage by Elasticsearch is
> almost three times the size of the index.
> store.type" : "memory"
> gateway.type: fs
>
> --
--
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [September 4, 2012, 7:53am UTC](https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766/5 "2012-09-04T07:53:21Z")
Hi Jalal,
please note that store type "memory" is not the size of the index as you
would expect from a simple cache, it uses a ByteBuffer directory where each
file is segmented into many buffers.
Best regards,
Jörg
--
---
### Author:  [@Jalal](https://discuss.elastic.co/u/Jalal)
#### Post date: [September 17, 2012, 12:49pm UTC](https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766/6 "2012-09-17T12:49:04Z")
Thanks ,
So it's better to use something like warmer API, right?
On Tue, Sep 4, 2012 at 1:23 PM, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote:
> Hi Jalal,
>
> please note that store type "memory" is not the size of the index as you
> would expect from a simple cache, it uses a ByteBuffer directory where each
> file is segmented into many buffers.
>
> Best regards,
>
> Jörg
>
> --
--
---
### Author:  [@jprante](https://discuss.elastic.co/u/jprante)
#### Post date: [September 17, 2012, 5:24pm UTC](https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766/7 "2012-09-17T17:24:52Z")
Yes, the warmer API is great for the case you have queries that run only
fast with all documents or values in memory. When a node just comes up, the
caches are not filled and queries tend to be slow. Instead of filling the
cache by the queries coming in, and user experiencing slow response times,
you can use the warmer API to reduce the numbers of slow queries in the
beginning of a node's lifetime.
Best regards,
Jörg
On Monday, September 17, 2012 2:49:07 PM UTC+2, Jalal wrote:
> Thanks ,
>
> So it's better to use something like warmer API, right?
>
> On Tue, Sep 4, 2012 at 1:23 PM, Jörg Prante \<[joerg...@gmail.com](mailto:joerg...@gmail.com)\
>
> > wrote:
>
> > Hi Jalal,
> >
> > please note that store type "memory" is not the size of the index as you
> > would expect from a simple cache, it uses a ByteBuffer directory where each
> > file is segmented into many buffers.
> >
> > Best regards,
> >
> > Jörg
> >
> > --
--
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 3:12am UTC](https://discuss.elastic.co/t/how-to-move-the-whole-data-to-main-memory/8766/8 "2017-07-06T03:12:33Z")