# 100% CPU on 1 Node with JMeter Tests **URL:** \ **Category:** Elasticsearch\ **Created:** [June 19, 2014, 1:20am UTC](https://discuss.elastic.co/t/100-cpu-on-1-node-with-jmeter-tests/18195 "2014-06-19T01:20:58Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![sairam\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sairam_2/32/1475_2.png) [@sairam\_2](https://discuss.elastic.co/u/sairam_2)\ **Post date:** [June 19, 2014, 1:20am UTC](https://discuss.elastic.co/t/100-cpu-on-1-node-with-jmeter-tests/18195/1 "2014-06-19T01:20:58Z")
One out of 4 nodes always spikes to 100% CPU when we do some load tests using JMeter (50 Threads, 50 Loops) with any query (Match\_All, Filtered Query etc.,). That particular node has 3 Shards with 2 Primary Shards. The other nodes have less than 40% CPU on them at the same time. The heap is set at 30GB on all of them. This is the GIST for Hot Threads [https://gist.github.com/RobloxSai/9f040bbd5ab7b58f2b1d](https://gist.github.com/RobloxSai/9f040bbd5ab7b58f2b1d) when the Test was running. Is there anything else that can be done to improve the performance? The Query Response times jump to 5-8 seconds when the CPU is hammered. [https://lh3.googleusercontent.com/-EDnXAEg34cA/U6I5fb2zNOI/AAAAAAAAAB4/DqybJhq3Yhc/s1600/4+Nodes+Setup.png](https://lh3.googleusercontent.com/-EDnXAEg34cA/U6I5fb2zNOI/AAAAAAAAAB4/DqybJhq3Yhc/s1600/4+Nodes+Setup.png) I had previously posted the specs of the Servers on another thread [https://groups.google.com/forum/?utm\_medium=email&utm\_source=footer#!topic/elasticsearch/P1o\_4bVvECA](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/elasticsearch/P1o_4bVvECA). Here are the Server Specs: _Machine Specs:_ Processor: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz Number of CPU cores: 24 Number of Physical CPUs: 2 Installed RAM: [~256 GB Total] 128 GB 128 GB 16 MB Drive: Two 278GB SAS Drive configured in RAID 0 _OS:_ Arch: 64bit(x86\_64) OS Type: Linux Kernel: 2.6.32-431.5.1.el6.x86\_64 OS Version: Red Hat Enterprise Linux Server release 6.5 (Santiago) Java Version: Java 1.7.0\_51 (Java 7u51 x64 version for Linux). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4c80c557-c85d-4319-b7cc-ddd2aebdbd95%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4c80c557-c85d-4319-b7cc-ddd2aebdbd95%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![sairam\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sairam_2/32/1475_2.png) [@sairam\_2](https://discuss.elastic.co/u/sairam_2)\ **Post date:** [June 19, 2014, 5:40pm UTC](https://discuss.elastic.co/t/100-cpu-on-1-node-with-jmeter-tests/18195/2 "2014-06-19T17:40:45Z")
Bump On Wednesday, June 18, 2014 6:20:58 PM UTC-7, [sai...@roblox.com](mailto:sai...@roblox.com) wrote: > One out of 4 nodes always spikes to 100% CPU when we do some load tests > using JMeter (50 Threads, 50 Loops) with any query (Match\_All, Filtered > Query etc.,). That particular node has 3 Shards with 2 Primary Shards. The > other nodes have less than 40% CPU on them at the same time. The heap is > set at 30GB on all of them. This is the GIST for Hot Threads > [https://gist.github.com/RobloxSai/9f040bbd5ab7b58f2b1d](https://gist.github.com/RobloxSai/9f040bbd5ab7b58f2b1d) when the Test > was running. Is there anything else that can be done to improve the > performance? The Query Response times jump to 5-8 seconds when the CPU is > hammered. > > [https://lh3.googleusercontent.com/-EDnXAEg34cA/U6I5fb2zNOI/AAAAAAAAAB4/DqybJhq3Yhc/s1600/4+Nodes+Setup.png](https://lh3.googleusercontent.com/-EDnXAEg34cA/U6I5fb2zNOI/AAAAAAAAAB4/DqybJhq3Yhc/s1600/4+Nodes+Setup.png) > > I had previously posted the specs of the Servers on another thread > [https://groups.google.com/forum/?utm\_medium=email&utm\_source=footer#!topic/elasticsearch/P1o\_4bVvECA](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/elasticsearch/P1o_4bVvECA). > Here are the Server Specs: > _Machine Specs:_ > Processor: Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz > Number of CPU cores: 24 > Number of Physical CPUs: 2 > Installed RAM: [~256 GB Total] 128 GB 128 GB 16 MB > Drive: Two 278GB SAS Drive configured in > RAID 0 > _OS:_ > Arch: 64bit(x86\_64) > OS Type: Linux > Kernel: 2.6.32-431.5.1.el6.x86\_64 > OS Version: Red Hat Enterprise Linux Server release 6.5 > (Santiago) > Java Version: Java 1.7.0\_51 (Java 7u51 x64 version for > Linux). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/57ed23cc-4623-4434-b550-e21723980d1b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/57ed23cc-4623-4434-b550-e21723980d1b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Cedric\_Hourcade](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cedric_hourcade/32/1451_2.png) [@Cedric\_Hourcade](https://discuss.elastic.co/u/Cedric_Hourcade)\ **Post date:** [June 20, 2014, 8:27am UTC](https://discuss.elastic.co/t/100-cpu-on-1-node-with-jmeter-tests/18195/3 "2014-06-20T08:27:29Z")
Hello, It wouldn't surprise me if both Black Mamba and Slapstick were hitting 100%, they have more shards and have to handle more requests than the others nodes. But in your case it's only one node. First, are you http requests evenly spread over the 4 nodes? You could also check that all your shards are about the same size? To check if it's an hardware problem I would: - disable the shards rebalacing - stop the cluster - switch the whole data directories from Black Mamba and Slapstick - start the cluster and rerun the benchmark You'll then see if the problem comes from the 3 shards or the server itself. Cédric Hourcade [ced@wal.fr](mailto:ced@wal.fr) On Thu, Jun 19, 2014 at 7:40 PM, [sairam@roblox.com](mailto:sairam@roblox.com) wrote: > Bump > > On Wednesday, June 18, 2014 6:20:58 PM UTC-7, [sai...@roblox.com](mailto:sai...@roblox.com) wrote: > > > One out of 4 nodes always spikes to 100% CPU when we do some load tests > > using JMeter (50 Threads, 50 Loops) with any query (Match\_All, Filtered > > Query etc.,). That particular node has 3 Shards with 2 Primary Shards. The > > other nodes have less than 40% CPU on them at the same time. The heap is > > set at 30GB on all of them. This is the GIST for Hot Threads > > [https://gist.github.com/RobloxSai/9f040bbd5ab7b58f2b1d](https://gist.github.com/RobloxSai/9f040bbd5ab7b58f2b1d) when the Test > > was running. Is there anything else that can be done to improve the > > performance? The Query Response times jump to 5-8 seconds when the CPU is > > hammered. > > > > [https://lh3.googleusercontent.com/-EDnXAEg34cA/U6I5fb2zNOI/AAAAAAAAAB4/DqybJhq3Yhc/s1600/4+Nodes+Setup.png](https://lh3.googleusercontent.com/-EDnXAEg34cA/U6I5fb2zNOI/AAAAAAAAAB4/DqybJhq3Yhc/s1600/4+Nodes+Setup.png) > > > > I had previously posted the specs of the Servers on another thread > > [https://groups.google.com/forum/?utm\_medium=email&utm\_source=footer#!topic/elasticsearch/P1o\_4bVvECA](https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!topic/elasticsearch/P1o_4bVvECA). > > Here are the Server Specs: > > _Machine Specs:_ > > Processor: Intel(R) Xeon(R) CPU E5-2630 0 @ > > 2.30GHz > > Number of CPU cores: 24 > > Number of Physical CPUs: 2 > > Installed RAM: [~256 GB Total] 128 GB 128 GB 16 MB > > Drive: Two 278GB SAS Drive configured in > > RAID 0 > > _OS:_ > > Arch: 64bit(x86\_64) > > OS Type: Linux > > Kernel: 2.6.32-431.5.1.el6.x86\_64 > > OS Version: Red Hat Enterprise Linux Server release > > 6.5 (Santiago) > > Java Version: Java 1.7.0\_51 (Java 7u51 x64 version for > > Linux). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/57ed23cc-4623-4434-b550-e21723980d1b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/57ed23cc-4623-4434-b550-e21723980d1b%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/57ed23cc-4623-4434-b550-e21723980d1b%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/57ed23cc-4623-4434-b550-e21723980d1b%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJQxjPPCtwhWJtGbY6dCU\_mU6cyyfh3dgkLEW-0FW%3DH4Ki7LdQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJQxjPPCtwhWJtGbY6dCU_mU6cyyfh3dgkLEW-0FW%3DH4Ki7LdQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![gliniuslive](https://avatars.discourse-cdn.com/v4/letter/g/53a042/32.png) [@gliniuslive](https://discuss.elastic.co/u/gliniuslive)\ **Post date:** [January 11, 2015, 8:54am UTC](https://discuss.elastic.co/t/100-cpu-on-1-node-with-jmeter-tests/18195/4 "2015-01-11T08:54:04Z")
Perhaps it's due to JMeter hitting only one node instead of distrubuting the load across all 4. Since JMeter 2.12 a brand new test element is available to deal with ELB, CDN, DNS Load balancing, etc. Try adding a [DNS Cache Manager](http://jmeter.apache.org/usermanual/component_reference.html#DNS_Cache_Manager) to your test plan and ensure that requests are being equally distributed across all endpoint nodes. See [The DNS Cache Manager: The Right Way To Test Load Balanced Apps](http://blazemeter.com/blog/dns-cache-manager-right-way-test-load-balanced-apps) guide for more detailed explanation and configuration instructions. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:39am UTC](https://discuss.elastic.co/t/100-cpu-on-1-node-with-jmeter-tests/18195/5 "2017-07-06T00:39:43Z")