Since elastic search is a search engine, the amount of memory given to it would be less than the total data that is stored in search engine.
Is this assumption valid?
If we have 100 GB of Data can elastic be configured in such a way that the memory is limited say 10 GB but we need the ability to search all data. What would happen as the data increases to 200GB can elastic search still work with the same allocated 10 GB memory? Basically is there a combination of in-memory structures and in disk structures when the physical memory is not enough.
At present the following are the various memory terms that X-Pack shows under memory usage. Can light be shed on what levers are present to control these type of memory usage.
Elastic Search Memory Usage
Index Memory - Lucene 1
Stored Fields: Heap Memory Used by Stored Fields (E.g _source)
Doc Values: Heap memory used by Doc Values
Norms: Heap memory used by Norms (Normalization Factors for query-time, text scoring)
Index Memory - Lucene 2
Terms: Heap memory used by Terms (e.g text)
Points: Heap memory used by Points (e.g numbers, IPs and Geo Data)
Index Memory - Lucene 3
Fixed Bitsets: Head memory used by Fixed Bit Sets (e.g deeply nested documents)
Term Vectors: Heap memory used by Term Vectors
Version Maps: Heap memory used by Versioning (e.g updates and deletes)
Index Memory - Elastic Search
Query Cache : Heap memory used by Query Cache (e.g cached filters)
Request Cache : Heap memory used by Request Cache (E.g instant aggregations)
Field data : Heap memory used by Fielddata (e.g global ordinals or explicitly enabled field data on text fields).
Index Writer : Heap memory used by the Index Writer
Apart from reducing documents to save memory, can the rest of the levers be exposed so that we can control how elastic search uses memory for these types of memory quotas, if it exceed can it load some to disk and reload it again if the query is based on these types of memory.
Elasticsearch is not an in memory data store, so you don't need a 1:1, memory:data ratio.
Trying to optimise for something that may not be a problem is a waste of time. So my question is why are you asking all these questions? Are you having issues, that may be solved other ways?
As we load data to this index namely 10 GB, 20 GB ... 100 GB, the elastic node memory keeps increasing in any or all of the above mentioned buckets namely Stored Values, Terms, Fixed Bitsets. Eventually the Heap Size fills up namely 4.9 GB of 5 GB etc, then the problem starts, all the new ingestion are at extremely slow pace.
So how do we get the lever to make sure that we say only Keep X bytes for each memory bucket (Stored Values, Terms etc) in memory.
Even if we do keep it, how do we make sure something like Least Recently Used (LRU) is picked to remove from memory and store in disk.
The problem becomes compounded when we now have multiple indices in a Node and multiple Nodes having multiple indices.
Lets say we have 3 nodes Node1, Node2 and Node3
We have 3 indices Index1, Index 2 and Index 3 with with 1 replica and 5 shards.
We now have a node to shard distribution of
3 Nodes 15 Primary Shards and 15 Replica Shards a total of 30 shards / 3 nodes with each node having 10 shards each.
Now if one of the Nodes namely Node2 happens to have the shards of the indices Index 1 and 2 and the memory is full the rest of the indices ingestion suffers for this Index 1 on Node2. So not only is the lever required at the Node Level we also need the levers at the index level.
Note:
With X-Pack Monitoring we are not sure where the other half given to native Lucene goes namely the off heap goes. There is no track of the memory utilized in this space namely the 5 GB allocated to Lucene.
Elasticsearch already uses LRU in a number of caches. It takes indices that haven't been used in 5 minutes and marks them as inactive and removes certain data structures from the heap. There are no levers to manage things like you mention.
Yet you seem to be talking about theories here so I will ask again, are you having specific problems you can provide details on?
It's not allocated to Lucene, it's used by the OS for Lucene files. You'd need to monitor your OS for details into that.
If there are no levers to control what is stored in memory. The only option is to wait for the 5 minute LRU cache inactive then the original assumption that we could store any amount of data into the same memory allocated to elastic search is flawed if all the indices are used at the same time.
This mean, that elastic search would need to have the memory increased as more and more data is ingested into the system.
Is this the same when the system data is at rest, i.e when there are no records being added, for the existing records too the memory can't be controlled leading to memory required even at rest.
Not sure I understand your theory comment. This is design with abstraction, if we substitute Index1, Index2 with tweet and customer indices you get your practical problem. Let me know if you would prefer specifics as opposed to abstraction.
Are you asking these questions because you have a specific problem you are solving, or are you just asking how Elasticsearch handles memory allocation?
I am having the exact problem which I have described which is as more and more data is ingested into the multiple indices present across multiple nodes. We allocate X memory to elastic nodes, we load Y data till the memory X peaks after Y, we increase memory to X' to elastic nodes and then we continue to load Y' data.
So want to understand if this is the expected design of memory management in elastic or anything else is available to control this behavior. Does it use the disk at all when the memory is full or can we force it to use disk when memory is full.
How can we expect to size this system if we are not sure what would be the memory requirement when all the data is loaded both for current and future needs or to say we could only afford X memory but we want Elastic to work only with X memory. Unlike Disk which are expandable, the memory is not as there are standard sizes, to increase memory most cloud providers also expect you to increase CPU which compounds the cost.
Elasticsearch will use any memory you provide. It does that to make performance as optimal as possible. But if you aren't seeing OOM or problems, then it's doing as it designed. If you don't add more heap then it'll manage that accordingly.
If you have a specific problem then providing specifics about that would be best. If you are seeing increasing heap use, what does that mean to your cluster? What exactly is "memory peaks" to you? What is your heap size? What is your infrastructure? What is the use pattern? Are you using Monitoring to check things? Are you OOMing? What are the GC cycles like? What version of Elasticsearch are you on? What JVM? What OS? What do your mappings look like? What queries are you running? How are you sending data to Elasticsearch? How much data is that? How many indices and shards? What are your future volume estimates?
This conversation started with Elastic search memory management is not efficient in managing the memory it was provided or work with constrained memory when the volume of data is not proportionate to the memory provided . There are not many levers or details present to control and influence how to manage the memory provided to it.
Do you have specific "how to" or articles on how to find the usage for any OS that the development team uses. This question has been asked many times but all answer don't have specifics. How do we find how much memory is used by the OS in relation to the root process of Elasticsearch? Basically what is the basis of recommendation to reserve 50% and how do we validate it? Namely is 50% up to the max of 32 GB memory which is 16 GB for Heap and 16 GB to OS. Or Max heap of 32 GB which would mean a 64 GB System that would give 32 GB for Heap and 32 GB for OS. At large memory allocation for OS, we should be able to identify if this is actually being used than just a trust statement.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
