We had several cases of OOM errors and I had thought it was related to
indexing or searching, but then noticed even when ES was idle (no
indexing or searching), an OOM would occur after 30-40 minutes of
starting up.
I am using 8g min/max heap size, all other settings on default. Index
size is 125gb in 200m docs per day.
When I change the max_merged_segment to 2g (from default of 5g), the
OOM errors do not occur. We have the default setting of 10 segments,
though each newly created index has between 60-70 segments after a 24-
hour period, and auto-merge reduces that to about 50 segments after 48-
hours. We create a index every 24 hours and rarely write to a
previous index.
Yes, merging is a process that requires memory, and can tip the memory usage
to cause OOM.
Regarding the number of segments, they are not relevant with the new tiered
based merge policy used.
On Tue, Aug 23, 2011 at 11:31 PM, Tom Le dottom@gmail.com wrote:
We had several cases of OOM errors and I had thought it was related to
indexing or searching, but then noticed even when ES was idle (no
indexing or searching), an OOM would occur after 30-40 minutes of
starting up.
I am using 8g min/max heap size, all other settings on default. Index
size is 125gb in 200m docs per day.
When I change the max_merged_segment to 2g (from default of 5g), the
OOM errors do not occur. We have the default setting of 10 segments,
though each newly created index has between 60-70 segments after a 24-
hour period, and auto-merge reduces that to about 50 segments after 48-
hours. We create a index every 24 hours and rarely write to a
previous index.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.