# Upgrade to ES 1.4 memory issues / tuning max\_merged\_segment **URL:** \ **Category:** Elasticsearch\ **Created:** [October 28, 2014, 3:41pm UTC](https://discuss.elastic.co/t/upgrade-to-es-1-4-memory-issues-tuning-max-merged-segment/20464 "2014-10-28T15:41:49Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![dzaebst](https://avatars.discourse-cdn.com/v4/letter/d/f0a364/32.png) [@dzaebst](https://discuss.elastic.co/u/dzaebst)\ **Post date:** [October 28, 2014, 3:41pm UTC](https://discuss.elastic.co/t/upgrade-to-es-1-4-memory-issues-tuning-max-merged-segment/20464/1 "2014-10-28T15:41:49Z")
Hi all, I have been testing an upgrade to elasticsearch 1.4 beta1. We use the Bulk API along with scripts to perform upserts into elasticsearch. These perform well under ES 1.2 without any tuning. However, in ES 1.4 beta1, running these upsert scripts often lead to: java.lang.OutOfMemoryError: Java heap space We use the bulk API: curl -iL -silent --show-error -XPOST 'localhost:9200/\_bulk' --data-binary @./\ where the file contains about 130 Mb ( 10,000 to 250,000 lines ) of data. It is filled with update / script commands: {"update":{"\_index":"2762\_2014\_41","\_type":"event","\_id":"97bc142e15c7136ebe866890e03dfad9"}} {"doc": {"type":"event","date\_time":"2014-10-17T19:00:00Z","day":20141017,"impression\_cost":0.005,"format":"xyz","impression":1,"referer":"xyz","browser":"xyz","os":"android 4.4.4","device":"nexus 4","channel":"mobile","x\_name":"xyz","id":"97bc142e15c7136ebe866890e03dfad9" },"doc\_as\_upsert":true } {"update":{"\_index":"2762\_2014\_41","\_type":"event","\_id":"97bc142e15c7136ebe866890e03dfad9"}} { "script":"if( ctx.\_source.containsKey("impression") ){ ctx.\_source.impression += 2; } else { ctx.\_source.impression = 2; };" } There were some issues with with permgen taking up memory in this ticket that have been addressed since the beta1 release, so we re-built from the 1.4 branch: > And I found this discussion about an OOM error that suggested including the max\_merged\_segment in elasticsearch.yml. [https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/max\_merged\_segment/elasticsearch/ETjvBVUvCJs/ZccfzUIFAKoJ](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/max_merged_segment/elasticsearch/ETjvBVUvCJs/ZccfzUIFAKoJ) index.merge.policy.max\_merged\_segment: 1gb Setting max\_merged\_segment, launching on my development machine with a 2gb: ES\_HEAP\_SIZE=2g ./bin/elasticsearch, and bringing down the file size per-bulk request to about 25Mb stablilzed the system. However, it would still heap dump when larger files like 130Mb were allowed. I don't fully understand how this fixed the memory issues. Would anyone be able to provide some insight into why we would run into memory issues with the upgrade? I'd like to better understand how the memory is managed here so that I can support this in production. Are there recommended sizes for bulk requests? And how those related to the max\_merged\_segment size? Thanks, Dave -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/d5845815-eb21-41c0-b899-96626dce577e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d5845815-eb21-41c0-b899-96626dce577e%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 28, 2014, 11:04pm UTC](https://discuss.elastic.co/t/upgrade-to-es-1-4-memory-issues-tuning-max-merged-segment/20464/2 "2014-10-28T23:04:12Z")
There are several areas of memory Elasticsearch is using when receiving large bulks over HTTP: - Netty buffers (HTTP chunking etc.) - bulk source (the lines are split into portions for each primary shard) - memory for analyzing/tokenizing the fields in the source - translog buffer (ES write ahead logging) - indexing buffer (Lucene NRT etc.) The longer the bulk runs, the more competition is for the 2g heap. If you run sustaining bulk requests for some time (say 15 - 20 minutes), ES picks up the created segments on disk and merges the segments to larger ones to keep the performance. Reducing the default of 5g to 1g for max\_merge\_segments has two effects. It allows for faster completion of a merge step because the volume of merge segment is limited, and it takes off some of the pressure on the heap when segments grow larger and larger. The downside is that merge steps are executed more frequently. You are correct, bulk requests around 1-10MB should work ok for most of the servers. Bulk requests of 100MB and larger have strong effects on the run time and the memory consumption for the other ES processing steps which are necessary to index the data, and should be reduced in order to find a "sweet spot" - the exact point of the optimal balance between bulk request input and indexing power depends also on other factors, like I/O throughput and CPU (plus ES settings like store throttling). Jörg On Tue, Oct 28, 2014 at 4:41 PM, [dzaebst@runads.com](mailto:dzaebst@runads.com) wrote: > Hi all, > > I have been testing an upgrade to elasticsearch 1.4 beta1. > > We use the Bulk API along with scripts to perform upserts into > elasticsearch. These perform well under ES 1.2 without any tuning. > > However, in ES 1.4 beta1, running these upsert scripts often lead to: > java.lang.OutOfMemoryError: Java heap space > > We use the bulk API: > > curl -iL -silent --show-error -XPOST 'localhost:9200/\_bulk' > --data-binary @./\ > > where the file contains about 130 Mb ( 10,000 to 250,000 lines ) of data. > It is filled with update / script commands: > > {"update":{"\_index":"2762\_2014\_41","\_type":"event","\_id":"97bc142e15c7136ebe866890e03dfad9"}} > {"doc": > > {"type":"event","date\_time":"2014-10-17T19:00:00Z","day":20141017,"impression\_cost":0.005,"format":"xyz","impression":1,"referer":"xyz","browser":"xyz","os":"android > 4.4.4","device":"nexus > 4","channel":"mobile","x\_name":"xyz","id":"97bc142e15c7136ebe866890e03dfad9" > },"doc\_as\_upsert":true > } > > {"update":{"\_index":"2762\_2014\_41","\_type":"event","\_id":"97bc142e15c7136ebe866890e03dfad9"}} > { > "script":"if( ctx.\_source.containsKey("impression") ){ > ctx.\_source.impression += 2; } else { ctx.\_source.impression = 2; };" > } > > There were some issues with with permgen taking up memory in this ticket > that have been addressed since the beta1 release, so we re-built from the > 1.4 branch: > [Reduce permgen use from Groovy scripts · Issue #7658 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/7658) > > And I found this discussion about an OOM error that suggested including > the max\_merged\_segment in elasticsearch.yml. > > [Redirecting to Google Groups](https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/max_merged_segment/elasticsearch/ETjvBVUvCJs/ZccfzUIFAKoJ) > > index.merge.policy.max\_merged\_segment: 1gb > > Setting max\_merged\_segment, launching on my development machine with a > 2gb: ES\_HEAP\_SIZE=2g ./bin/elasticsearch, and bringing down the file size > per-bulk request to about 25Mb stablilzed the system. > However, it would still heap dump when larger files like 130Mb were > allowed. > > I don't fully understand how this fixed the memory issues. Would anyone > be able to provide some insight into why we would run into memory issues > with the upgrade? > I'd like to better understand how the memory is managed here so that I can > support this in production. Are there recommended sizes for bulk > requests? And how those related to the max\_merged\_segment size? > > Thanks, > Dave > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/d5845815-eb21-41c0-b899-96626dce577e%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/d5845815-eb21-41c0-b899-96626dce577e%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/d5845815-eb21-41c0-b899-96626dce577e%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/d5845815-eb21-41c0-b899-96626dce577e%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGqFR\_QSBMKiynb%2BpbLKh-VvEoGzj8iJiHv5VL41QKZDA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGqFR_QSBMKiynb%2BpbLKh-VvEoGzj8iJiHv5VL41QKZDA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:53am UTC](https://discuss.elastic.co/t/upgrade-to-es-1-4-memory-issues-tuning-max-merged-segment/20464/3 "2017-07-06T00:53:19Z")