# Very frequent ES OOM's & potential segment merge problems **URL:** \ **Category:** Elasticsearch\ **Created:** [June 19, 2014, 8:35pm UTC](https://discuss.elastic.co/t/very-frequent-es-ooms-potential-segment-merge-problems/18220 "2014-06-19T20:35:28Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Paul\_Sabou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sabou/32/1472_2.png) [@Paul\_Sabou](https://discuss.elastic.co/u/Paul_Sabou)\ **Post date:** [June 19, 2014, 8:35pm UTC](https://discuss.elastic.co/t/very-frequent-es-ooms-potential-segment-merge-problems/18220/1 "2014-06-19T20:35:28Z")
Hi, _Situation:_ We are using ES 1.2.1 on a machine with 32GB RAM, fast SSD and 12 cores. The machine runs Ubuntu 14.0.x LTS. The ES process has 12GB of RAM allocated. We have an index in which we inserted 105 million small documents so the ES data folder is around 50GB in size (we see this by using du -h . on the folder) The new document insertion rate is rather small (ie. 100-300 small docs per second). _The problem:_ We experienced rather frequent ES OOM (Out of Memory) at a rate of around one every 15 mins. To lower the load on the index we deleted 104+ million docs (ie. mostly small log entries) by deleting everything in one type : curl -XDELETE [http://localhost:9200/index\_xx/type\_yy](http://localhost:9200/index_xx/type_yy) so that we ended up with an ES index with several thousands docs. After this we started to experience massive disk IO (10-20Mbs reads and 1MBs writes) and more frequent OOM's (at a rate of around one every 7 minutes). We restart ES after every OOM and kept monitoring the data folder size. Over the next hour the size went down to around 36GB but now it's stuck there (doesn't go down in size even after several hours). _Questions_ : Is this a problem related to segment merging running out of memory? If so how can be solved? If not, what could be the problem? Thanks Paul. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/695c92a3-f77a-46bd-9041-79421a0bf1be%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/695c92a3-f77a-46bd-9041-79421a0bf1be%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [June 20, 2014, 7:37am UTC](https://discuss.elastic.co/t/very-frequent-es-ooms-potential-segment-merge-problems/18220/2 "2014-06-20T07:37:57Z")
Hey, can you provide more information about the OOM exception? Also you should use the nodes stats API to monitor your system, so you can maybe easily spot, where this memory consumption stems from. Also, are you just indexing or doing searches/queries/gets as well? --Alex On Thu, Jun 19, 2014 at 10:35 PM, Paul Sabou [paul.sabou@gmail.com](mailto:paul.sabou@gmail.com) wrote: > Hi, > > _Situation:_ > We are using ES 1.2.1 on a machine with 32GB RAM, fast SSD and 12 cores. The > machine runs Ubuntu 14.0.x LTS. > The ES process has 12GB of RAM allocated. > > We have an index in which we inserted 105 million small documents so the > ES data folder is around 50GB in size > (we see this by using du -h . on the folder) > > The new document insertion rate is rather small (ie. 100-300 small docs > per second). > > _The problem:_ > > We experienced rather frequent ES OOM (Out of Memory) at a rate of around > one every 15 mins. To lower the load on the index > we deleted 104+ million docs (ie. mostly small log entries) by deleting > everything in one type : > curl -XDELETE [http://localhost:9200/index\_xx/type\_yy](http://localhost:9200/index_xx/type_yy) > > so that we ended up with an ES index with several thousands docs. > After this we started to experience massive disk IO (10-20Mbs reads and > 1MBs writes) and more frequent OOM's (at a rate of around > one every 7 minutes). We restart ES after every OOM and kept monitoring > the data folder size. Over the next hour the size went down > to around 36GB but now it's stuck there (doesn't go down in size even > after several hours). > > _Questions_ : > Is this a problem related to segment merging running out of memory? If so > how can be solved? > If not, what could be the problem? > > Thanks > Paul. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/695c92a3-f77a-46bd-9041-79421a0bf1be%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/695c92a3-f77a-46bd-9041-79421a0bf1be%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/695c92a3-f77a-46bd-9041-79421a0bf1be%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/695c92a3-f77a-46bd-9041-79421a0bf1be%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8Ed84KwzVg1MTK8Da83YgO6pjb3QMLVwCT%2B48NPw3HfA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8Ed84KwzVg1MTK8Da83YgO6pjb3QMLVwCT%2B48NPw3HfA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Paul\_Sabou](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sabou/32/1472_2.png) [@Paul\_Sabou](https://discuss.elastic.co/u/Paul_Sabou)\ **Post date:** [June 20, 2014, 8:19am UTC](https://discuss.elastic.co/t/very-frequent-es-ooms-potential-segment-merge-problems/18220/3 "2014-06-20T08:19:20Z")
java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot complete merge at org.apache.lucene.index.IndexWriter.commitMerge(IndexWriter.java:3546) at org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4272) at org.apache.lucene.index.IndexWriter.merge(IndexWriter.java:3728) at org.apache.lucene.index.ConcurrentMergeScheduler.doMerge(ConcurrentMergeScheduler.java:405) at org.apache.lucene.index.TrackingConcurrentMergeScheduler.doMerge(TrackingConcurrentMergeScheduler.java:106) at org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:482) On Thursday, June 19, 2014 10:35:28 PM UTC+2, Paul Sabou wrote: > Hi, > > _Situation:_ > We are using ES 1.2.1 on a machine with 32GB RAM, fast SSD and 12 cores. The > machine runs Ubuntu 14.0.x LTS. > The ES process has 12GB of RAM allocated. > > We have an index in which we inserted 105 million small documents so the > ES data folder is around 50GB in size > (we see this by using du -h . on the folder) > > The new document insertion rate is rather small (ie. 100-300 small docs > per second). > > _The problem:_ > > We experienced rather frequent ES OOM (Out of Memory) at a rate of around > one every 15 mins. To lower the load on the index > we deleted 104+ million docs (ie. mostly small log entries) by deleting > everything in one type : > curl -XDELETE [http://localhost:9200/index\_xx/type\_yy](http://localhost:9200/index_xx/type_yy) > > so that we ended up with an ES index with several thousands docs. > After this we started to experience massive disk IO (10-20Mbs reads and > 1MBs writes) and more frequent OOM's (at a rate of around > one every 7 minutes). We restart ES after every OOM and kept monitoring > the data folder size. Over the next hour the size went down > to around 36GB but now it's stuck there (doesn't go down in size even > after several hours). > > _Questions_ : > Is this a problem related to segment merging running out of memory? If so > how can be solved? > If not, what could be the problem? > > Thanks > Paul. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/db4e6c34-2d6b-4623-aa9c-c6fbf9083ea9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/db4e6c34-2d6b-4623-aa9c-c6fbf9083ea9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:20am UTC](https://discuss.elastic.co/t/very-frequent-es-ooms-potential-segment-merge-problems/18220/4 "2017-07-06T01:20:59Z")