# How to improve the cluster's perfomance for the JVM Out of Memory Exception? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 15, 2012, 1:25pm UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703 "2012-05-15T13:25:34Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![manoj\_2](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@manoj\_2](https://discuss.elastic.co/u/manoj_2)\ **Post date:** [May 15, 2012, 1:25pm UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703/1 "2012-05-15T13:25:34Z")
Hi Everyone, We face performance issue(JVM: Out of memory Exception) when faceting a huge index data. We tried moving from Array Type to String Type of the fields which are taken for faceting, still of no improvement. I would like to know if we are using the current Cluster machines to the Max efficiency We face a performance issue with the current ES configuration. We have 2 server machines in cluster, each with below configurations... RAM:128 GB Index data are in DISK: RAID 5 -\>7200rpm Processor:24 Core processor ES Configuration: JVM MAX MEM:16GB(remaining of RAM is unused now) Total Index Size:850GB In another thread where kimchy answered to increase the JVM to overcome the exception. How much can I increase for the current machines' configuration. Is thr a need of new cluster for this case. Also, I am just interested to know the MAX efficiency that could be pulled out of a single cluster.... Please let me know if any more info I need to provide for deciding up on this.... Thanks a lot! Manoj ---
**Author:** ![manoj\_2](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@manoj\_2](https://discuss.elastic.co/u/manoj_2)\ **Post date:** [May 16, 2012, 6:36am UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703/2 "2012-05-16T06:36:17Z")
Kimchy, any information about this? Really expecting to know the realization behind the scenes... On Tuesday, May 15, 2012 6:55:34 PM UTC+5:30, Manoj wrote: > Hi Everyone, > > We face performance issue(JVM: Out of memory Exception) when faceting a > huge index data. We tried moving from Array Type to String Type of the > fields which are taken for faceting, still of no improvement. I would like > to know if we are using the current Cluster machines to the Max efficiency > > We face a performance issue with the current ES configuration. We have 2 > server machines in cluster, each with below configurations... > > RAM:128 GB > Index data are in DISK: RAID 5 -\>7200rpm > Processor:24 Core processor > > ES Configuration: > JVM MAX MEM:16GB(remaining of RAM is unused now) > Total Index Size:850GB > > In another thread where kimchy answered to increase the JVM to overcome > the exception. How much can I increase for the current machines' > configuration. Is thr a need of new cluster for this case. > > Also, I am just interested to know the MAX efficiency that could be pulled > out of a single cluster.... > > Please let me know if any more info I need to provide for deciding up on > this.... > > Thanks a lot! > Manoj ---
**Author:** ![jschelle\_2](https://avatars.discourse-cdn.com/v4/letter/j/f475e1/32.png) [@jschelle\_2](https://discuss.elastic.co/u/jschelle_2)\ **Post date:** [May 16, 2012, 2:46pm UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703/3 "2012-05-16T14:46:44Z")
I can't seem to find it at the moment, but there was another post where kimchy mentions that performance seriously degrades if elasticsearch starts paging. Since you seem to have a lot of unused RAM, you might try using the rest. Also, what time of performance are you trying to optimize, search, filter, percolate?? Depending on what you're doing there are a lot of settings to play with. On Wednesday, May 16, 2012 2:36:17 AM UTC-4, Manoj wrote: > Kimchy, any information about this? > > Really expecting to know the realization behind the scenes... > > On Tuesday, May 15, 2012 6:55:34 PM UTC+5:30, Manoj wrote: > > > Hi Everyone, > > > > We face performance issue(JVM: Out of memory Exception) when faceting a > > huge index data. We tried moving from Array Type to String Type of the > > fields which are taken for faceting, still of no improvement. I would like > > to know if we are using the current Cluster machines to the Max efficiency > > > > We face a performance issue with the current ES configuration. We have 2 > > server machines in cluster, each with below configurations... > > > > RAM:128 GB > > Index data are in DISK: RAID 5 -\>7200rpm > > Processor:24 Core processor > > > > ES Configuration: > > JVM MAX MEM:16GB(remaining of RAM is unused now) > > Total Index Size:850GB > > > > In another thread where kimchy answered to increase the JVM to overcome > > the exception. How much can I increase for the current machines' > > configuration. Is thr a need of new cluster for this case. > > > > Also, I am just interested to know the MAX efficiency that could be > > pulled out of a single cluster.... > > > > Please let me know if any more info I need to provide for deciding up on > > this.... > > > > Thanks a lot! > > Manoj ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [May 16, 2012, 2:58pm UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703/4 "2012-05-16T14:58:13Z")
You can dedicate a lot more memory to ES JVM. Typically you can dedicate half of the physical memory. So you can try 64GB, or even more if you still experience problems. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Tue, May 15, 2012 at 9:25 AM, Manoj [manokrrish@gmail.com](mailto:manokrrish@gmail.com) wrote: > Hi Everyone, > > We face performance issue(JVM: Out of memory Exception) when faceting a > huge index data. We tried moving from Array Type to String Type of the > fields which are taken for faceting, still of no improvement. I would like > to know if we are using the current Cluster machines to the Max efficiency > > We face a performance issue with the current ES configuration. We have 2 > server machines in cluster, each with below configurations... > > RAM:128 GB > Index data are in DISK: RAID 5 -\>7200rpm > Processor:24 Core processor > > ES Configuration: > JVM MAX MEM:16GB(remaining of RAM is unused now) > Total Index Size:850GB > > In another thread where kimchy answered to increase the JVM to overcome > the exception. How much can I increase for the current machines' > configuration. Is thr a need of new cluster for this case. > > Also, I am just interested to know the MAX efficiency that could be pulled > out of a single cluster.... > > Please let me know if any more info I need to provide for deciding up on > this.... > > Thanks a lot! > Manoj ---
**Author:** ![manoj\_2](https://avatars.discourse-cdn.com/v4/letter/m/e0b2c6/32.png) [@manoj\_2](https://discuss.elastic.co/u/manoj_2)\ **Post date:** [May 17, 2012, 4:40am UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703/5 "2012-05-17T04:40:41Z")
Thanks for the reply shadow000fire ... We are requiring more Physical Memory when use Faceting on a field which has a lot of Tag kind of data. Unless we do this Faceting everything goes fine with the current RAM. But we are in need of the feature at present. Also I want to ask u this... Assume the faceting of the particular field requires 64GB to work fine for a single instance of search, does it mean, it require the multiples of the same size, if we increase the concurrent number of users?(ie. 128GB RAM for 2 concurrent search, etc..) Is there any more optimization I need to do in ES settings wise, for the situation we face? On Wednesday, May 16, 2012 8:16:44 PM UTC+5:30, shadow000fire wrote: > I can't seem to find it at the moment, but there was another post where > kimchy mentions that performance seriously degrades if elasticsearch starts > paging. Since you seem to have a lot of unused RAM, you might try using > the rest. Also, what time of performance are you trying to optimize, > search, filter, percolate?? Depending on what you're doing there are a lot > of settings to play with. > > On Wednesday, May 16, 2012 2:36:17 AM UTC-4, Manoj wrote: > > > Kimchy, any information about this? > > > > Really expecting to know the realization behind the scenes... > > > > On Tuesday, May 15, 2012 6:55:34 PM UTC+5:30, Manoj wrote: > > > > > Hi Everyone, > > > > > > We face performance issue(JVM: Out of memory Exception) when faceting a > > > huge index data. We tried moving from Array Type to String Type of the > > > fields which are taken for faceting, still of no improvement. I would like > > > to know if we are using the current Cluster machines to the Max efficiency > > > > > > We face a performance issue with the current ES configuration. We have > > > 2 server machines in cluster, each with below configurations... > > > > > > RAM:128 GB > > > Index data are in DISK: RAID 5 -\>7200rpm > > > Processor:24 Core processor > > > > > > ES Configuration: > > > JVM MAX MEM:16GB(remaining of RAM is unused now) > > > Total Index Size:850GB > > > > > > In another thread where kimchy answered to increase the JVM to overcome > > > the exception. How much can I increase for the current machines' > > > configuration. Is thr a need of new cluster for this case. > > > > > > Also, I am just interested to know the MAX efficiency that could be > > > pulled out of a single cluster.... > > > > > > Please let me know if any more info I need to provide for deciding up on > > > this.... > > > > > > Thanks a lot! > > > Manoj ---
**Author:** ![Andy\_Wick](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andy_wick/32/44017_2.png) [@Andy\_Wick](https://discuss.elastic.co/u/Andy_Wick)\ **Post date:** [May 17, 2012, 12:58pm UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703/6 "2012-05-17T12:58:43Z")
I've been facing the same issue and here is what I think I know and what I've done. Would love corrections or other peoples experiences. 🙂 Facet memory usage is determined by the number of documents, size of the field, and if using multi value the max number of values in a single field. Using jvisualvm it appears almost all the memory is int arrays, which I think are the ordinals arrays being passed around everywhere. It seems like for memory reduction switching from arrays to sparse arrays might help, but that is a Kimchy question, and would probably hurt performance. What I've done to reduce memory: - Disable \_all since I don't need it ("\_all" : {"enabled" : false}) - Enable \_source compression ("\_source" : {"compress" : true}) - Enable soft cache, which seems to make the process run longer before hitting OOM at the cost of slow queries. Since OOMs are evil this is a ok thing, just use bigdesk to see how much more memory you need. ("index.cache.field.type" : "soft") - Turn off replication ("index.number\_of\_replicas" : 0) - Only have 1 or 2 shards per node - Make sure the shards are evenly distributed ("index.routing.allocation.total\_shards\_per\_node" : 1 (or 2)) - If using string facets make sure the mapping index is "not\_analyzed" - I switched from string tags to short tags, with a separate index/type that has the string-\>short conversion. I create the conversion on the fly using the \_version technique of yet another index/type. ( See [http://blogs.perl.org/users/clinton\_gormley/2011/10/elasticsearchsequence---a-blazing-fast-ticket-server.html](http://blogs.perl.org/users/clinton_gormley/2011/10/elasticsearchsequence---a-blazing-fast-ticket-server.html)) - Since I'm using multi value shorts I reduced the max number of tags per document which really helped. - Switched to \_bulk indexing - Lowered my refresh interval ("index.refresh\_interval" : 60) For your situation you should definitely give elastic more memory. I would be tempted to run 3 or more 20G-30G nodes on the 128G machine to help spread out the GC pauses. (If using linux with multi nodes per machine, don't use mlockall unless you have a 3.x kernel.) Although it seems multi node vs single huge node hasn't been proven yet. I'm about to switch to 64G machines and I'm going to be trying 2x20G nodes per machine. You definitely want to leave some free memory for the disk cache, the "free" command on linux shows what is currently being used in the cached column. Thanks, Andy ---
**Author:** ![Mohammady\_Mahdy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohammady_mahdy/32/2128_2.png) [@Mohammady\_Mahdy](https://discuss.elastic.co/u/Mohammady_Mahdy)\ **Post date:** [January 24, 2013, 9:33am UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703/7 "2013-01-24T09:33:25Z")
@Andy just want to pick your brain about the single huge node vs multiple smaller nodes tradeoff and how did it work for you. Anybody else watching this thread. Your feedback is highly appreciated. On Thursday, May 17, 2012 4:58:43 PM UTC+4, Andy Wick wrote: > I've been facing the same issue and here is what I think I know and what > I've done. Would love corrections or other peoples experiences. 🙂 > > Facet memory usage is determined by the number of documents, size of the > field, and if using multi value the max number of values in a single field. > Using jvisualvm it appears almost all the memory is int arrays, which I > think are the ordinals arrays being passed around everywhere. It seems > like for memory reduction switching from arrays to sparse arrays might > help, but that is a Kimchy question, and would probably hurt performance. > > What I've done to reduce memory: > > - Disable \_all since I don't need it ("\_all" : {"enabled" : false}) > - Enable \_source compression ("\_source" : {"compress" : true}) > - Enable soft cache, which seems to make the process run longer before > hitting OOM at the cost of slow queries. Since OOMs are evil this is a ok > thing, just use bigdesk to see how much more memory you need. > ("index.cache.field.type" : "soft") > - Turn off replication ("index.number\_of\_replicas" : 0) > - Only have 1 or 2 shards per node > - Make sure the shards are evenly distributed > ("index.routing.allocation.total\_shards\_per\_node" : 1 (or 2)) > - If using string facets make sure the mapping index is "not\_analyzed" > - I switched from string tags to short tags, with a separate index/type > that has the string-\>short conversion. I create the conversion on the fly > using the \_version technique of yet another index/type. ( See > [ElasticSearch::Sequence - a blazing fast ticket server | Clinton Gormley [blogs.perl.org]](http://blogs.perl.org/users/clinton_gormley/2011/10/elasticsearchsequence---a-blazing-fast-ticket-server.html) > ) > - Since I'm using multi value shorts I reduced the max number of tags > per document which really helped. > - Switched to \_bulk indexing > - Lowered my refresh interval ("index.refresh\_interval" : 60) > > For your situation you should definitely give elastic more memory. I > would be tempted to run 3 or more 20G-30G nodes on the 128G machine to help > spread out the GC pauses. (If using linux with multi nodes per machine, > don't use mlockall unless you have a 3.x kernel.) Although it seems multi > node vs single huge node hasn't been proven yet. I'm about to switch to > 64G machines and I'm going to be trying 2x20G nodes per machine. You > definitely want to leave some free memory for the disk cache, the "free" > command on linux shows what is currently being used in the cached column. > > Thanks, > Andy -- ---
**Author:** ![Dan\_Fairs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_fairs/32/1205_2.png) [@Dan\_Fairs](https://discuss.elastic.co/u/Dan_Fairs)\ **Post date:** [January 24, 2013, 1:51pm UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703/8 "2013-01-24T13:51:29Z")
> - Lowered my refresh interval ("index.refresh\_interval" : 60) Just a quick note - it appears that the default unit for refresh\_interval, if you don't specify one, is milliseconds. We ran our cluster for a short while with a rather more frequent refresh\_interval than we wanted! ## Cheers, Dan Dan Fairs | [dan.fairs@gmail.com](mailto:dan.fairs@gmail.com) | @danfairs | [secondsync.com](http://secondsync.com) -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:54am UTC](https://discuss.elastic.co/t/how-to-improve-the-clusters-perfomance-for-the-jvm-out-of-memory-exception/7703/9 "2017-07-06T02:54:45Z")