# Periodic CPU spikes **URL:** \ **Category:** Elasticsearch\ **Created:** [February 1, 2012, 11:43pm UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569 "2012-02-01T23:43:27Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Hao\_Lian](https://avatars.discourse-cdn.com/v4/letter/h/dec6dc/32.png) [@Hao\_Lian](https://discuss.elastic.co/u/Hao_Lian)\ **Post date:** [February 1, 2012, 11:43pm UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/1 "2012-02-01T23:43:27Z")
Hi! We've been load-testing Elastic Search before we release to production (and we love it!) and we've been noticing periodic CPU spikes: [http://i.imgur.com/PIbNb.png](http://i.imgur.com/PIbNb.png) We have two indices, both with the same mapping: [https://raw.github.com/gist/f959b5f1502e12e1dc2b/c95c208ff9c6232de5846ea6abd4b9f96c8c6570/gistfile1.txt](https://raw.github.com/gist/f959b5f1502e12e1dc2b/c95c208ff9c6232de5846ea6abd4b9f96c8c6570/gistfile1.txt) The only APIs we use are the Index and the Search API. We used to have long-request problems because we'd send Search requests with 1000+ facets but we rewrote that code recently to use one facet (histogram) and it's been smooth sailing then. However, the CPU spikes persist and every now and then one of our calls to ES will time out. (We use NEST the C# client to talk to ES.) We're running ES on an eight-gigabyte, four-core virtual machine. Here are our node stats: [https://raw.github.com/gist/f0246240e0285ca16bfb/d1090e3cbaf7347b55f5e5356bf61caff072ba56/gistfile1.txt](https://raw.github.com/gist/f0246240e0285ca16bfb/d1090e3cbaf7347b55f5e5356bf61caff072ba56/gistfile1.txt) and our node health: [https://raw.github.com/gist/b4edaa73c14be8829919/431138ed026b576a2ea7362b879c75dde337576c/gistfile1.txt](https://raw.github.com/gist/b4edaa73c14be8829919/431138ed026b576a2ea7362b879c75dde337576c/gistfile1.txt) It's using the standard configuration (1 node, 1 replica, 10 shards total) that comes out of the box, and we're running ES via the service script and conf file that came with ES. What do you guys think we should do to make these spikes go away? Should we be running more nodes (if so, a node on a different box or a node on the same box)? Any insight into this problem would be super-appreciated. And I don't know if this is relevant at all, but the only way I've been able to simulate a spike is by POSTing to \_cache/clear. (That's why you see three spikes back-to-back in that graph toward the end; that's me, poking around, sending cache-clear POSTs.) Thanks! Hao Lian Fog Creek Software ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [February 2, 2012, 5:32am UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/2 "2012-02-02T05:32:03Z")
Hao, This could be JVM GC in action. Do you have something to monitor the JVMs? I'd point you to our Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) - but it currently supports only Solr, not Elasticsearch yet. Try using jstat, even that will tell you if it's GC. ## Otis > **[Sematext | IT System Monitoring Tools for DevOps](https://sematext.com/)** > > IT system monitoring and management tools for DevOps who need 24x7 live visibility into their infrastructure. On Feb 1, 6:43 pm, Hao Lian [h...@fogcreek.com](mailto:h...@fogcreek.com) wrote: > Hi! > > We've been load-testing Elastic Search before we release to production > (and we love it!) and we've been noticing periodic CPU spikes: > > [Imgur: The magic of the Internet](http://i.imgur.com/PIbNb.png) > > We have two indices, both with the same mapping: > > [https://raw.github.com/gist/f959b5f1502e12e1dc2b/c95c208ff9c6232de584](https://raw.github.com/gist/f959b5f1502e12e1dc2b/c95c208ff9c6232de584)... > > The only APIs we use are the Index and the Search API. We used to have > long-request problems because we'd send Search requests with 1000+ > facets but we rewrote that code recently to use one facet (histogram) > and it's been smooth sailing then. However, the CPU spikes persist and > every now and then one of our calls to ES will time out. (We use NEST > the C# client to talk to ES.) > > We're running ES on an eight-gigabyte, four-core virtual machine. Here > are our node stats: > > [https://raw.github.com/gist/f0246240e0285ca16bfb/d1090e3cbaf7347b55f5](https://raw.github.com/gist/f0246240e0285ca16bfb/d1090e3cbaf7347b55f5)... > > and our node health: > > [https://raw.github.com/gist/b4edaa73c14be8829919/431138ed026b576a2ea7](https://raw.github.com/gist/b4edaa73c14be8829919/431138ed026b576a2ea7)... > > It's using the standard configuration (1 node, 1 replica, 10 shards > total) that comes out of the box, and we're running ES via the service > script and conf file that came with ES. What do you guys think we > should do to make these spikes go away? Should we be running more > nodes (if so, a node on a different box or a node on the same box)? > Any insight into this problem would be super-appreciated. > > And I don't know if this is relevant at all, but the only way I've > been able to simulate a spike is by POSTing to \_cache/clear. (That's > why you see three spikes back-to-back in that graph toward the end; > that's me, poking around, sending cache-clear POSTs.) > > Thanks! > Hao Lian > Fog Creek Software ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 2, 2012, 7:05pm UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/3 "2012-02-02T19:05:44Z")
Heya, You sent the node info and not node stats, can you send the node stats as well? From the node info, I can already see something things that you should do: 1. Set the min and max memory to be the same. And set to ~4gb out of the machine total 8gb. 2. Use a more recent JVM, the one you are using is old. 3. Are you running on windows or \*nix? If \*unix, enable mlockall (so the java process will not swap). 4. Install big desk, and see what happens when the CPU spikes, does the GC time / count increases around the same interval? 5. It might be internal merges going on, you can tell if merges are happening using the node stats API, and how long they took (hopefully we will have this graphed in big desk soon). On Thursday, February 2, 2012 at 1:43 AM, Hao Lian wrote: > Hi! > > We've been load-testing Elastic Search before we release to production > (and we love it!) and we've been noticing periodic CPU spikes: > > [http://i.imgur.com/PIbNb.png](http://i.imgur.com/PIbNb.png) > > We have two indices, both with the same mapping: > > [https://raw.github.com/gist/f959b5f1502e12e1dc2b/c95c208ff9c6232de5846ea6abd4b9f96c8c6570/gistfile1.txt](https://raw.github.com/gist/f959b5f1502e12e1dc2b/c95c208ff9c6232de5846ea6abd4b9f96c8c6570/gistfile1.txt) > > The only APIs we use are the Index and the Search API. We used to have > long-request problems because we'd send Search requests with 1000+ > facets but we rewrote that code recently to use one facet (histogram) > and it's been smooth sailing then. However, the CPU spikes persist and > every now and then one of our calls to ES will time out. (We use NEST > the C# client to talk to ES.) > > We're running ES on an eight-gigabyte, four-core virtual machine. Here > are our node stats: > > [https://raw.github.com/gist/f0246240e0285ca16bfb/d1090e3cbaf7347b55f5e5356bf61caff072ba56/gistfile1.txt](https://raw.github.com/gist/f0246240e0285ca16bfb/d1090e3cbaf7347b55f5e5356bf61caff072ba56/gistfile1.txt) > > and our node health: > > [https://raw.github.com/gist/b4edaa73c14be8829919/431138ed026b576a2ea7362b879c75dde337576c/gistfile1.txt](https://raw.github.com/gist/b4edaa73c14be8829919/431138ed026b576a2ea7362b879c75dde337576c/gistfile1.txt) > > It's using the standard configuration (1 node, 1 replica, 10 shards > total) that comes out of the box, and we're running ES via the service > script and conf file that came with ES. What do you guys think we > should do to make these spikes go away? Should we be running more > nodes (if so, a node on a different box or a node on the same box)? > Any insight into this problem would be super-appreciated. > > And I don't know if this is relevant at all, but the only way I've > been able to simulate a spike is by POSTing to \_cache/clear. (That's > why you see three spikes back-to-back in that graph toward the end; > that's me, poking around, sending cache-clear POSTs.) > > Thanks! > Hao Lian > Fog Creek Software ---
**Author:** ![Hao\_Lian](https://avatars.discourse-cdn.com/v4/letter/h/dec6dc/32.png) [@Hao\_Lian](https://discuss.elastic.co/u/Hao_Lian)\ **Post date:** [February 2, 2012, 7:43pm UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/4 "2012-02-02T19:43:51Z")
Thanks Otis and Shay, we'll get right on following those leads. Here are our node stats, sorry about that mixup: [https://raw.github.com/gist/78972f7ed681bfb44a24/b365f854df4a5df427c9352ed6afef225ad3fd7a/gistfile1.txt](https://raw.github.com/gist/78972f7ed681bfb44a24/b365f854df4a5df427c9352ed6afef225ad3fd7a/gistfile1.txt) (non-spiky load) [https://raw.github.com/gist/ff0a02b9dcff4b84c554/367695337c9113f6e66e735fa81d0985e6858540/gistfile1.txt](https://raw.github.com/gist/ff0a02b9dcff4b84c554/367695337c9113f6e66e735fa81d0985e6858540/gistfile1.txt) (spiky load, pulled from yesterday) The merges don't look too problematic in either case, but thanks for the heads up there. We are running on Debian Linux; investigating mlockall as we speak in addition to installing big desk and jstat. Thanks again! Hao. ---
**Author:** ![Lukas\_Vlcek1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lukas_vlcek1/32/819_2.png) [@Lukas\_Vlcek1](https://discuss.elastic.co/u/Lukas_Vlcek1)\ **Post date:** [February 2, 2012, 8:09pm UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/5 "2012-02-02T20:09:43Z")
Not sure which ES version you are using, so just note about bigdesk, it does not support 0.19.0 snapshot ES version very well, there are a lot of breaking changes in ES master, but I am working on a fix as we speak... On Thu, Feb 2, 2012 at 8:43 PM, Hao Lian [hao@fogcreek.com](mailto:hao@fogcreek.com) wrote: > Thanks Otis and Shay, we'll get right on following those leads. > > Here are our node stats, sorry about that mixup: > > [https://raw.github.com/gist/78972f7ed681bfb44a24/b365f854df4a5df427c9352ed6afef225ad3fd7a/gistfile1.txt](https://raw.github.com/gist/78972f7ed681bfb44a24/b365f854df4a5df427c9352ed6afef225ad3fd7a/gistfile1.txt) > (non-spiky load) > > [https://raw.github.com/gist/ff0a02b9dcff4b84c554/367695337c9113f6e66e735fa81d0985e6858540/gistfile1.txt](https://raw.github.com/gist/ff0a02b9dcff4b84c554/367695337c9113f6e66e735fa81d0985e6858540/gistfile1.txt) > (spiky load, pulled from yesterday) > > The merges don't look too problematic in either case, but thanks for > the heads up there. > > We are running on Debian Linux; investigating mlockall as we speak in > addition to installing big desk and jstat. > > Thanks again! > Hao. ---
**Author:** ![Hao\_Lian](https://avatars.discourse-cdn.com/v4/letter/h/dec6dc/32.png) [@Hao\_Lian](https://discuss.elastic.co/u/Hao_Lian)\ **Post date:** [February 3, 2012, 5:21pm UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/6 "2012-02-03T17:21:56Z")
OK, we think we fixed it, and I just wanted to report a little about what we did. We couldn't upgrade to JVM 7 because it would've been too big a drain on our sysadmin resources, but we did solve the problem. - Setting ES\_MIN\_MEM = ES\_MAX\_MEM helped substantially in that they made it much easier to see what was going on with the Java GC. - BigDesk is awesome! Way better than just running curl over and over. - jstat is awesome! - It turns out that ES on SurvivorRatio=8 with UseParNewGC would slowly fill up eden space until the GC paused and moved those objects to S0 or S1. This was causing all our CPU spikes because our eden space was 2 GB given that we have a 6 GB heap and the default old:new ratio is 1:2. Once we realized this, we bounded the eden space with NewSize and MaxNewSize to be 320 MB (down from 2 GB). Now GC happens more periodically but faster, saturating at most one of our cores rather than all four. We've got more tuning to do, but for now ES remains responsive during these smaller but more frequent collections, which makes us happy. Thanks for everybody's help! Hao Lian Fog Creek Software ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 5, 2012, 1:35pm UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/7 "2012-02-05T13:35:10Z")
Heya, - You don't have to move to Java 7, but can you try and run with a newer Java 6 version? u18 is very old. - Where did you see that the value of the new generation was 2gb? Can you gist it? On Friday, February 3, 2012 at 7:21 PM, Hao Lian wrote: > OK, we think we fixed it, and I just wanted to report a little about > what we did. We couldn't upgrade to JVM 7 because it would've been too > big a drain on our sysadmin resources, but we did solve the problem. > > - Setting ES\_MIN\_MEM = ES\_MAX\_MEM helped substantially in that they > made it much easier to see what was going on with the Java GC. > > - BigDesk is awesome! Way better than just running curl over and over. > > - jstat is awesome! > > - It turns out that ES on SurvivorRatio=8 with UseParNewGC would > slowly fill up eden space until the GC paused and moved those objects > to S0 or S1. This was causing all our CPU spikes because our eden > space was 2 GB given that we have a 6 GB heap and the default old:new > ratio is 1:2. Once we realized this, we bounded the eden space with > NewSize and MaxNewSize to be 320 MB (down from 2 GB). Now GC happens > more periodically but faster, saturating at most one of our cores > rather than all four. We've got more tuning to do, but for now ES > remains responsive during these smaller but more frequent collections, > which makes us happy. > > Thanks for everybody's help! > Hao Lian > Fog Creek Software ---
**Author:** ![Hao\_Lian](https://avatars.discourse-cdn.com/v4/letter/h/dec6dc/32.png) [@Hao\_Lian](https://discuss.elastic.co/u/Hao_Lian)\ **Post date:** [February 6, 2012, 4:15pm UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/8 "2012-02-06T16:15:33Z")
> You don't have to move to Java 7, but can you try and run with a newer Java 6 version? u18 is very old. Good idea! I'll ask our sysadmins. > Where did you see that the value of the new generation was 2gb? We calculated this purely from JVM docs. (If there's a tool to measure this for us, we'd love to have it.) I'm redoing the calculations and getting different values now 😃 so I think it's entirely possible we made an error in the heat of the moment, but our impetus for decreasing it was that we were seeing eden steadily climb up and up in jstat until it became full at which point a move to the survivor spaces occurred and spiked the CPU. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 7, 2012, 10:09am UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/9 "2012-02-07T10:09:06Z")
I added support to get the memory pool sizes when getting the JVM stats using the node stats API, so it will be simpler to know what values each has (including the eden/new). On Monday, February 6, 2012 at 6:15 PM, Hao Lian wrote: > > You don't have to move to Java 7, but can you try and run with a newer Java 6 version? u18 is very old. > > Good idea! I'll ask our sysadmins. > > > Where did you see that the value of the new generation was 2gb? > > We calculated this purely from JVM docs. (If there's a tool to measure > this for us, we'd love to have it.) I'm redoing the calculations and > getting different values now 😃 so I think it's entirely possible we > made an error in the heat of the moment, but our impetus for > decreasing it was that we were seeing eden steadily climb up and up in > jstat until it became full at which point a move to the survivor > spaces occurred and spiked the CPU. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:40am UTC](https://discuss.elastic.co/t/periodic-cpu-spikes/6569/10 "2017-07-06T03:40:21Z")