First going to assume you mean 8GBs of memory or I am very impressed that
ElasticSearch runs at all.
Second, when are you running out of memory?
Do you run out of memory while indexing?
Is it a specific document when indexing?
Do you run out of memory when searching?
Is it a specific search when searching?
What type of search, sort, filter?
How many documents do you index each day
What is the largest document?
What is the average document?
Are you indexing in batches?
How big are your batches?
Of your 8 gb how much is allocated to ElasticSearch?
How much is allocated to File System Cache?
(I usually start with 2 GB to the OS, and split the remaining ram
between ElasticSearch and FileSystem Cache. This means allocate 3GB to
By a rough swag based on the very little info you have provided, I would
say that your cluster does not have enough ram for the level of data you
are trying to load into it. In general I have found that lucene indexes
like to be in memory. When they cannot performance is poor and operations
can fail. By indexing 100GBs of data a day, you are asking ElasticSearch
to store some pretty large segments for 8GB or memory (effectively 3GB of
I ran into similar problems with machines that had only 8GB or memory when
indexing. My data volume was lower than what you have indicated.
Upgrading to larger instances with 16GB resolved the issue and I have not
had a problem since. Of course I had tuned everything previously according
to what I outlined above. The 16 GB box means that instead of 3GB for ES
you have (16G-2G)/2= 7GB, more than double. In consulting engagements I
always recommend 16GB as a bare minimum, but 32GB as a realistic minimum.
This page also has some good info on it:
On Thursday, March 12, 2015 at 6:12:11 PM UTC-6, Jeferson Martins wrote:
I have 5 nodes of ElasticSearch with 4 CPUs, 8 Mbs of RAM.
My Index today have 1TB of data and my index have about 100GBs By day and
i configure 3 primary shards and 1 replica but my elasticsearch gets
OutOfMemoy in every two days.
There is some configuration to resolve this problem?