Very frequent ES OOM's & potential segment merge problems

Elastic Stack Elasticsearch
1

Hi,

Situation:
We are using ES 1.2.1 on a machine with 32GB RAM, fast SSD and 12 cores. The
machine runs Ubuntu 14.0.x LTS.
The ES process has 12GB of RAM allocated.

We have an index in which we inserted 105 million small documents so the ES
data folder is around 50GB in size
(we see this by using du -h . on the folder)

The new document insertion rate is rather small (ie. 100-300 small docs per
second).

The problem:

We experienced rather frequent ES OOM (Out of Memory) at a rate of around
one every 15 mins. To lower the load on the index
we deleted 104+ million docs (ie. mostly small log entries) by deleting
everything in one type :
curl -XDELETE http://localhost:9200/index_xx/type_yy

so that we ended up with an ES index with several thousands docs.
After this we started to experience massive disk IO (10-20Mbs reads and
1MBs writes) and more frequent OOM's (at a rate of around
one every 7 minutes). We restart ES after every OOM and kept monitoring the
data folder size. Over the next hour the size went down
to around 36GB but now it's stuck there (doesn't go down in size even after
several hours).

Questions :
Is this a problem related to segment merging running out of memory? If so
how can be solved?
If not, what could be the problem?

Thanks
Paul.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/695c92a3-f77a-46bd-9041-79421a0bf1be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hey,

can you provide more information about the OOM exception? Also you should
use the nodes stats API to monitor your system, so you can maybe easily
spot, where this memory consumption stems from. Also, are you just indexing
or doing searches/queries/gets as well?

--Alex

On Thu, Jun 19, 2014 at 10:35 PM, Paul Sabou paul.sabou@gmail.com wrote:

Hi,

Situation:
We are using ES 1.2.1 on a machine with 32GB RAM, fast SSD and 12 cores. The
machine runs Ubuntu 14.0.x LTS.
The ES process has 12GB of RAM allocated.

We have an index in which we inserted 105 million small documents so the
ES data folder is around 50GB in size
(we see this by using du -h . on the folder)

The new document insertion rate is rather small (ie. 100-300 small docs
per second).

The problem:

We experienced rather frequent ES OOM (Out of Memory) at a rate of around
one every 15 mins. To lower the load on the index
we deleted 104+ million docs (ie. mostly small log entries) by deleting
everything in one type :
curl -XDELETE http://localhost:9200/index_xx/type_yy

so that we ended up with an ES index with several thousands docs.
After this we started to experience massive disk IO (10-20Mbs reads and
1MBs writes) and more frequent OOM's (at a rate of around
one every 7 minutes). We restart ES after every OOM and kept monitoring
the data folder size. Over the next hour the size went down
to around 36GB but now it's stuck there (doesn't go down in size even
after several hours).

Questions :
Is this a problem related to segment merging running out of memory? If so
how can be solved?
If not, what could be the problem?

Thanks
Paul.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/695c92a3-f77a-46bd-9041-79421a0bf1be%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/695c92a3-f77a-46bd-9041-79421a0bf1be%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8Ed84KwzVg1MTK8Da83YgO6pjb3QMLVwCT%2B48NPw3HfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

java.lang.IllegalStateException: this writer hit an OutOfMemoryError;
cannot complete merge
at
org.apache.lucene.index.IndexWriter.commitMerge(IndexWriter.java:3546)
at
org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4272)
at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3728)
at
org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405)
at
org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:106)
at
org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482)

On Thursday, June 19, 2014 10:35:28 PM UTC+2, Paul Sabou wrote:

Hi,

Situation:
We are using ES 1.2.1 on a machine with 32GB RAM, fast SSD and 12 cores. The
machine runs Ubuntu 14.0.x LTS.
The ES process has 12GB of RAM allocated.

We have an index in which we inserted 105 million small documents so the
ES data folder is around 50GB in size
(we see this by using du -h . on the folder)

The new document insertion rate is rather small (ie. 100-300 small docs
per second).

The problem:

We experienced rather frequent ES OOM (Out of Memory) at a rate of around
one every 15 mins. To lower the load on the index
we deleted 104+ million docs (ie. mostly small log entries) by deleting
everything in one type :
curl -XDELETE http://localhost:9200/index_xx/type_yy

so that we ended up with an ES index with several thousands docs.
After this we started to experience massive disk IO (10-20Mbs reads and
1MBs writes) and more frequent OOM's (at a rate of around
one every 7 minutes). We restart ES after every OOM and kept monitoring
the data folder size. Over the next hour the size went down
to around 36GB but now it's stuck there (doesn't go down in size even
after several hours).

Questions :
Is this a problem related to segment merging running out of memory? If so
how can be solved?
If not, what could be the problem?

Thanks
Paul.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/db4e6c34-2d6b-4623-aa9c-c6fbf9083ea9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.