Hello ES group,
I have had recurring heap problems ("java.lang.OutOfMemoryError: Java heap
space”) on my 2-nodes ES cluster (16GB RAM/node, 8GB allocated to ES) the
last month and I really don’t know how to tackle them.
It started at a time where I was doing aggregations on a “milliseconds
since EPOCH” field, and I was given to understand that it was probably the
cause of my problems since it created a very large number of buckets before
aggregating them. So I stopped doing aggregations on this field (I did not
delete it though).
Recently I was told that my index had too few shards respective to its size
(2 primary shards, 1 replica each, 100-150 Mdocs). So I decided to try
reindexing into a new index with more shards (I am using es-reindex.rb,
which itself uses the bulk API). But now I am having OutOfMemoryError happen
during reindexing. Needless to say, once an OutOfMemoryError happens, my
cluster seems to never recover until I reboot each node.
It should be noted that I use ES almost exclusively with search_type=count,
since I am only trying to do analytics on website data.
I am not sure how to proceed from this point, I don’t know the right tool
to pinpoint my memory problems and there doesn’t seem to be a way to ask ES
for heap usage by index/query/task type
I’d be very grateful for any advice you can offer.
Thanks in advance,
Vincent Bernardi
In my situation I had configured ES_HEAP_SIZE in the yml file with no
effectt (on centos 6.5)
ES was running with default memory configured allways. This is evenso with
ES 1.3.4.
My solution was to configure this in /etc/init.d/elasticsearch
Another setting that can be helpfull:
indices.fielddata.cache.size: 40%
On Wednesday, October 22, 2014 11:19:34 AM UTC+2, Vincent Bernardi wrote:
Hello ES group,
I have had recurring heap problems ("java.lang.OutOfMemoryError: Java
heap space”) on my 2-nodes ES cluster (16GB RAM/node, 8GB allocated to
ES) the last month and I really don’t know how to tackle them.
It started at a time where I was doing aggregations on a “milliseconds
since EPOCH” field, and I was given to understand that it was probably the
cause of my problems since it created a very large number of buckets before
aggregating them. So I stopped doing aggregations on this field (I did not
delete it though).
Recently I was told that my index had too few shards respective to its
size (2 primary shards, 1 replica each, 100-150 Mdocs). So I decided to try
reindexing into a new index with more shards (I am using es-reindex.rb,
which itself uses the bulk API). But now I am having OutOfMemoryError happen
during reindexing. Needless to say, once an OutOfMemoryError happens, my
cluster seems to never recover until I reboot each node.
It should be noted that I use ES almost exclusively with
search_type=count, since I am only trying to do analytics on website data.
I am not sure how to proceed from this point, I don’t know the right tool
to pinpoint my memory problems and there doesn’t seem to be a way to ask ES
for heap usage by index/query/task type
I’d be very grateful for any advice you can offer.
Thanks in advance,
Vincent Bernardi
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.