# What levers are present to control memory usage in Elastic Search? **URL:** \ **Category:** Elasticsearch\ **Created:** [November 28, 2017, 11:34pm UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494 "2017-11-28T23:34:43Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![antonpious](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonpious/32/31885_2.png) [@antonpious](https://discuss.elastic.co/u/antonpious)\ **Post date:** [November 28, 2017, 11:34pm UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494/1 "2017-11-28T23:34:43Z")
Since elastic search is a search engine, the amount of memory given to it would be less than the total data that is stored in search engine. Is this assumption valid? If we have 100 GB of Data can elastic be configured in such a way that the memory is limited say 10 GB but we need the ability to search all data. What would happen as the data increases to 200GB can elastic search still work with the same allocated 10 GB memory? Basically is there a combination of in-memory structures and in disk structures when the physical memory is not enough. At present the following are the various memory terms that X-Pack shows under memory usage. Can light be shed on what levers are present to control these type of memory usage. Elastic Search Memory Usage **Index Memory - Lucene 1** Stored Fields: Heap Memory Used by Stored Fields (E.g \_source) Doc Values: Heap memory used by Doc Values Norms: Heap memory used by Norms (Normalization Factors for query-time, text scoring) **Index Memory - Lucene 2** Terms: Heap memory used by Terms (e.g text) Points: Heap memory used by Points (e.g numbers, IPs and Geo Data) **Index Memory - Lucene 3** Fixed Bitsets: Head memory used by Fixed Bit Sets (e.g deeply nested documents) Term Vectors: Heap memory used by Term Vectors Version Maps: Heap memory used by Versioning (e.g updates and deletes) **Index Memory - Elastic Search** Query Cache : Heap memory used by Query Cache (e.g cached filters) Request Cache : Heap memory used by Request Cache (E.g instant aggregations) Field data : Heap memory used by Fielddata (e.g global ordinals or explicitly enabled field data on text fields). Index Writer : Heap memory used by the Index Writer Apart from reducing documents to save memory, can the rest of the levers be exposed so that we can control how elastic search uses memory for these types of memory quotas, if it exceed can it load some to disk and reload it again if the query is based on these types of memory. I found one article on how to control the field data . [https://www.elastic.co/guide/en/elasticsearch/reference/6.0/modules-fielddata.html](https://www.elastic.co/guide/en/elasticsearch/reference/6.0/modules-fielddata.html) Can the corresponding lever documentation be exposed or pointed to find how to limit memory usage? specifically Terms and Fixed Bitsets. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 29, 2017, 2:08am UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494/2 "2017-11-29T02:08:45Z")
Elasticsearch is not an in memory data store, so you don't need a 1:1, memory:data ratio. Trying to optimise for something that may not be a problem is a waste of time. So my question is why are you asking all these questions? Are you having issues, that may be solved other ways? ---
**Author:** ![antonpious](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonpious/32/31885_2.png) [@antonpious](https://discuss.elastic.co/u/antonpious)\ **Post date:** [November 29, 2017, 4:31am UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494/3 "2017-11-29T04:31:10Z")
Once we have confirmed that its not a 1:1 Memory:Data ratio the problem becomes obvious. Lets say we have 1 Node with 1 Index with 10 GB memory of which half is given to the JVM namely 5 GB and another half for Lucene which is 5 GB. As recommended in "Give less than Half your memory to Lucene" ([https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html](https://www.elastic.co/guide/en/elasticsearch/guide/current/heap-sizing.html)) As we load data to this index namely 10 GB, 20 GB ... 100 GB, the elastic node memory keeps increasing in any or all of the above mentioned buckets namely Stored Values, Terms, Fixed Bitsets. Eventually the Heap Size fills up namely 4.9 GB of 5 GB etc, then the problem starts, all the new ingestion are at extremely slow pace. So how do we get the lever to make sure that we say only Keep X bytes for each memory bucket (Stored Values, Terms etc) in memory. Even if we do keep it, how do we make sure something like Least Recently Used (LRU) is picked to remove from memory and store in disk. The problem becomes compounded when we now have multiple indices in a Node and multiple Nodes having multiple indices. Lets say we have 3 nodes Node1, Node2 and Node3 We have 3 indices Index1, Index 2 and Index 3 with with 1 replica and 5 shards. We now have a node to shard distribution of 3 Nodes 15 Primary Shards and 15 Replica Shards a total of 30 shards / 3 nodes with each node having 10 shards each. Now if one of the Nodes namely Node2 happens to have the shards of the indices Index 1 and 2 and the memory is full the rest of the indices ingestion suffers for this Index 1 on Node2. So not only is the lever required at the Node Level we also need the levers at the index level. Note: With X-Pack Monitoring we are not sure where the other half given to native Lucene goes namely the off heap goes. There is no track of the memory utilized in this space namely the 5 GB allocated to Lucene. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 29, 2017, 5:33am UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494/4 "2017-11-29T05:33:44Z")
Elasticsearch already uses LRU in a number of caches. It takes indices that haven't been used in 5 minutes and marks them as inactive and removes certain data structures from the heap. There are no levers to manage things like you mention. Yet you seem to be talking about theories here so I will ask again, are you having specific problems you can provide details on? > [@antonpious](#): > > There is no track of the memory utilized in this space namely the 5 GB allocated to Lucene. It's not allocated to Lucene, it's used by the OS for Lucene files. You'd need to monitor your OS for details into that. ---
**Author:** ![antonpious](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonpious/32/31885_2.png) [@antonpious](https://discuss.elastic.co/u/antonpious)\ **Post date:** [November 29, 2017, 6:11am UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494/5 "2017-11-29T06:11:47Z")
If there are no levers to control what is stored in memory. The only option is to wait for the 5 minute LRU cache inactive then the original assumption that we could store any amount of data into the same memory allocated to elastic search is flawed if all the indices are used at the same time. This mean, that elastic search would need to have the memory increased as more and more data is ingested into the system. Is this the same when the system data is at rest, i.e when there are no records being added, for the existing records too the memory can't be controlled leading to memory required even at rest. Not sure I understand your theory comment. This is design with abstraction, if we substitute Index1, Index2 with tweet and customer indices you get your practical problem. Let me know if you would prefer specifics as opposed to abstraction. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 29, 2017, 6:13am UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494/6 "2017-11-29T06:13:21Z")
Are you asking these questions because you have a specific problem you are solving, or are you just asking how Elasticsearch handles memory allocation? ---
**Author:** ![antonpious](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonpious/32/31885_2.png) [@antonpious](https://discuss.elastic.co/u/antonpious)\ **Post date:** [November 29, 2017, 6:24am UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494/7 "2017-11-29T06:24:15Z")
I am having the exact problem which I have described which is as more and more data is ingested into the multiple indices present across multiple nodes. We allocate X memory to elastic nodes, we load Y data till the memory X peaks after Y, we increase memory to X' to elastic nodes and then we continue to load Y' data. So want to understand if this is the expected design of memory management in elastic or anything else is available to control this behavior. Does it use the disk at all when the memory is full or can we force it to use disk when memory is full. How can we expect to size this system if we are not sure what would be the memory requirement when all the data is loaded both for current and future needs or to say we could only afford X memory but we want Elastic to work only with X memory. Unlike Disk which are expandable, the memory is not as there are standard sizes, to increase memory most cloud providers also expect you to increase CPU which compounds the cost. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 29, 2017, 7:39am UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494/8 "2017-11-29T07:39:00Z")
> [@antonpious](#): > > I am having the exact problem which I have described which is as more and more data is ingested into the multiple indices present across multiple nodes. We allocate X memory to elastic nodes, we load Y data till the memory X peaks after Y, we increase memory to X' to elastic nodes and then we continue to load Y' data. Elasticsearch will use any memory you provide. It does that to make performance as optimal as possible. But if you aren't seeing OOM or problems, then it's doing as it designed. If you don't add more heap then it'll manage that accordingly. If you have a specific problem then providing specifics about that would be best. If you are seeing increasing heap use, what does that mean to your cluster? What exactly is "memory peaks" to you? What is your heap size? What is your infrastructure? What is the use pattern? Are you using Monitoring to check things? Are you OOMing? What are the GC cycles like? What version of Elasticsearch are you on? What JVM? What OS? What do your mappings look like? What queries are you running? How are you sending data to Elasticsearch? How much data is that? How many indices and shards? What are your future volume estimates? ---
**Author:** ![antonpious](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/antonpious/32/31885_2.png) [@antonpious](https://discuss.elastic.co/u/antonpious)\ **Post date:** [December 1, 2017, 11:07pm UTC](https://discuss.elastic.co/t/what-levers-are-present-to-control-memory-usage-in-elastic-search/109494/9 "2017-12-01T23:07:00Z")
> [@warkolm](#): > > Elasticsearch will use any memory you provide. It does that to make performance as optimal as possible. But if you aren't seeing OOM or problems, then it's doing as it designed. If you don't add more heap then it'll manage that accordingly. This conversation started with Elastic search memory management is not efficient in managing the memory it was provided or work with constrained memory when the volume of data is not proportionate to the memory provided . There are not many levers or details present to control and influence how to manage the memory provided to it. I have raised the following feature requests > > > \