# Minor temp performance hit when JVM heap size is reached **URL:** \ **Category:** Elasticsearch\ **Created:** [November 3, 2010, 5:20pm UTC](https://discuss.elastic.co/t/minor-temp-performance-hit-when-jvm-heap-size-is-reached/3511 "2010-11-03T17:20:27Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [November 3, 2010, 5:20pm UTC](https://discuss.elastic.co/t/minor-temp-performance-hit-when-jvm-heap-size-is-reached/3511/1 "2010-11-03T17:20:27Z")
Hey, This isn't a huge deal, but wanted to point it out to see if there are any ways to avoid. Current production load is pretty low, 40 or so queries per second max. Performance is great and we know we can ramp up the traffic to 25x this rate. Average response times are ~25ms and max are ~200ms. However, we have seen two performance hits, where averages jumped to a sec and maxs up to 12 seconds. Both times this has occurred it directly correlates one of the machines in the cluster hitting its JVM heap max for elasticsearch. After the limit has been hit, each machine has been fine. Any ideas if there is anyway to avoid this hit? I'm guessing that the GC kicks on and ends up blocking some processing. Thanks, Paul ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 3, 2010, 5:37pm UTC](https://discuss.elastic.co/t/minor-temp-performance-hit-when-jvm-heap-size-is-reached/3511/2 "2010-11-03T17:37:58Z")
Yes, its basically the GC kicking in. One problem that might cause long GC is the mem being swapped to disk, causing the GC to be really slow since it needs to process the whole memory (thus loading the swapped memory). This will be better in 0.13 as I added support (copied form cassandra) to force the OS not to swap out the JVM. -shay.banon On Wed, Nov 3, 2010 at 7:20 PM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > Hey, > This isn't a huge deal, but wanted to point it out to see if there > are any ways to avoid. Current production load is pretty low, 40 or so > queries per second max. Performance is great and we know we can ramp > up the traffic to 25x this rate. Average response times are ~25ms and > max are ~200ms. > > However, we have seen two performance hits, where averages jumped to a > sec and maxs up to 12 seconds. Both times this has occurred it > directly correlates one of the machines in the cluster hitting its JVM > heap max for elasticsearch. After the limit has been hit, each machine > has been fine. > > Any ideas if there is anyway to avoid this hit? I'm guessing that the > GC kicks on and ends up blocking some processing. > > Thanks, > Paul ---
**Author:** ![tfreitas](https://avatars.discourse-cdn.com/v4/letter/t/bbe5ce/32.png) [@tfreitas](https://discuss.elastic.co/u/tfreitas)\ **Post date:** [November 3, 2010, 6:15pm UTC](https://discuss.elastic.co/t/minor-temp-performance-hit-when-jvm-heap-size-is-reached/3511/3 "2010-11-03T18:15:01Z")
Hi maybe this link helps to JVM tuning > **[JVM Performance Tuning](http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html)** > > Last week was JBoss World, and it was exciting to be a part of it. I gave a presentation on performance tuning our Enterprise Application P... Tony On Nov 3, 1:37 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Yes, its basically the GC kicking in. One problem that might cause long GC > is the mem being swapped to disk, causing the GC to be really slow since it > needs to process the whole memory (thus loading the swapped memory). This > will be better in 0.13 as I added support (copied form cassandra) to force > the OS not to swap out the JVM. > > -shay.banon > > On Wed, Nov 3, 2010 at 7:20 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > Hey, > > This isn't a huge deal, but wanted to point it out to see if there > > are any ways to avoid. Current production load is pretty low, 40 or so > > queries per second max. Performance is great and we know we can ramp > > up the traffic to 25x this rate. Average response times are ~25ms and > > max are ~200ms. > > > However, we have seen two performance hits, where averages jumped to a > > sec and maxs up to 12 seconds. Both times this has occurred it > > directly correlates one of the machines in the cluster hitting its JVM > > heap max for elasticsearch. After the limit has been hit, each machine > > has been fine. > > > Any ideas if there is anyway to avoid this hit? I'm guessing that the > > GC kicks on and ends up blocking some processing. > > > Thanks, > > Paul ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [November 3, 2010, 10:56pm UTC](https://discuss.elastic.co/t/minor-temp-performance-hit-when-jvm-heap-size-is-reached/3511/4 "2010-11-03T22:56:08Z")
Awesome. Great link. Like I said, not concerned, more curious. Will see how things look after moving to 0.13. Thanks, Paul On Nov 3, 12:15 pm, tfreitas [tfrei...@gmail.com](mailto:tfrei...@gmail.com) wrote: > Hi > > maybe this link helps to JVM tuning > > [Open Source and Enterprise Architecture: JVM Performance Tuning](http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html) > > Tony > > On Nov 3, 1:37 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Yes, its basically the GC kicking in. One problem that might cause long GC > > is the mem being swapped to disk, causing the GC to be really slow since it > > needs to process the whole memory (thus loading the swapped memory). This > > will be better in 0.13 as I added support (copied form cassandra) to force > > the OS not to swap out the JVM. > > > -shay.banon > > > On Wed, Nov 3, 2010 at 7:20 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > Hey, > > > This isn't a huge deal, but wanted to point it out to see if there > > > are any ways to avoid. Current production load is pretty low, 40 or so > > > queries per second max. Performance is great and we know we can ramp > > > up the traffic to 25x this rate. Average response times are ~25ms and > > > max are ~200ms. > > > > However, we have seen two performance hits, where averages jumped to a > > > sec and maxs up to 12 seconds. Both times this has occurred it > > > directly correlates one of the machines in the cluster hitting its JVM > > > heap max for elasticsearch. After the limit has been hit, each machine > > > has been fine. > > > > Any ideas if there is anyway to avoid this hit? I'm guessing that the > > > GC kicks on and ends up blocking some processing. > > > > Thanks, > > > Paul ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 3, 2010, 11:13pm UTC](https://discuss.elastic.co/t/minor-temp-performance-hit-when-jvm-heap-size-is-reached/3511/5 "2010-11-03T23:13:39Z")
Just one note regarding the 0.13 not swapping feature, it requires setting the minimum and the maximum memory allocation for the JVM to be the same. On Thu, Nov 4, 2010 at 12:56 AM, Paul [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > Awesome. Great link. > > Like I said, not concerned, more curious. Will see how things look > after moving to 0.13. > > Thanks, > Paul > > On Nov 3, 12:15 pm, tfreitas [tfrei...@gmail.com](mailto:tfrei...@gmail.com) wrote: > > > Hi > > > > maybe this link helps to JVM tuning > > > > [Open Source and Enterprise Architecture: JVM Performance Tuning](http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html) > > > > Tony > > > > On Nov 3, 1:37 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Yes, its basically the GC kicking in. One problem that might cause long > > > GC > > > is the mem being swapped to disk, causing the GC to be really slow > > > since it > > > needs to process the whole memory (thus loading the swapped memory). > > > This > > > will be better in 0.13 as I added support (copied form cassandra) to > > > force > > > the OS not to swap out the JVM. > > > > > -shay.banon > > > > > On Wed, Nov 3, 2010 at 7:20 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > > > Hey, > > > > This isn't a huge deal, but wanted to point it out to see if there > > > > are any ways to avoid. Current production load is pretty low, 40 or > > > > so > > > > queries per second max. Performance is great and we know we can ramp > > > > up the traffic to 25x this rate. Average response times are ~25ms and > > > > max are ~200ms. > > > > > > However, we have seen two performance hits, where averages jumped to > > > > a > > > > sec and maxs up to 12 seconds. Both times this has occurred it > > > > directly correlates one of the machines in the cluster hitting its > > > > JVM > > > > heap max for elasticsearch. After the limit has been hit, each > > > > machine > > > > has been fine. > > > > > > Any ideas if there is anyway to avoid this hit? I'm guessing that the > > > > GC kicks on and ends up blocking some processing. > > > > > > Thanks, > > > > Paul ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [December 2, 2010, 10:25pm UTC](https://discuss.elastic.co/t/minor-temp-performance-hit-when-jvm-heap-size-is-reached/3511/6 "2010-12-02T22:25:40Z")
Just wanted to say that 0.13.0 has been running for around a week, now and we haven't hit this case again. Really awesome work. Many thanks to Shay and the Cassandra team for figuring out how to solve this case. On Nov 3, 4:13 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Just one note regarding the 0.13 not swapping feature, it requires setting > the minimum and the maximum memory allocation for the JVM to be the same. > > On Thu, Nov 4, 2010 at 12:56 AM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > Awesome. Great link. > > > Like I said, not concerned, more curious. Will see how things look > > after moving to 0.13. > > > Thanks, > > Paul > > > On Nov 3, 12:15 pm, tfreitas [tfrei...@gmail.com](mailto:tfrei...@gmail.com) wrote: > > > > > Hi > > > > maybe this link helps to JVM tuning > > > > [Open Source and Enterprise Architecture: JVM Performance Tuning](http://andrigoss.blogspot.com/2008/02/jvm-performance-tuning.html) > > > > Tony > > > > On Nov 3, 1:37 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Yes, its basically the GC kicking in. One problem that might cause long > > > > GC > > > > is the mem being swapped to disk, causing the GC to be really slow > > > > since it > > > > needs to process the whole memory (thus loading the swapped memory). > > > > This > > > > will be better in 0.13 as I added support (copied form cassandra) to > > > > force > > > > the OS not to swap out the JVM. > > > > > -shay.banon > > > > > On Wed, Nov 3, 2010 at 7:20 PM, Paul [ppea...@gmail.com](mailto:ppea...@gmail.com) wrote: > > > > > > > > > Hey, > > > > > This isn't a huge deal, but wanted to point it out to see if there > > > > > are any ways to avoid. Current production load is pretty low, 40 or > > > > > so > > > > > queries per second max. Performance is great and we know we can ramp > > > > > up the traffic to 25x this rate. Average response times are ~25ms and > > > > > max are ~200ms. > > > > > > However, we have seen two performance hits, where averages jumped to > > > > > a > > > > > sec and maxs up to 12 seconds. Both times this has occurred it > > > > > directly correlates one of the machines in the cluster hitting its > > > > > JVM > > > > > heap max for elasticsearch. After the limit has been hit, each > > > > > machine > > > > > has been fine. > > > > > > Any ideas if there is anyway to avoid this hit? I'm guessing that the > > > > > GC kicks on and ends up blocking some processing. > > > > > > Thanks, > > > > > Paul ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:15am UTC](https://discuss.elastic.co/t/minor-temp-performance-hit-when-jvm-heap-size-is-reached/3511/7 "2017-07-06T04:15:48Z")
Closed on July 6, 2017, 4:15am UTC