# Master High CPU **URL:** \ **Category:** Elasticsearch\ **Created:** [April 7, 2014, 7:14am UTC](https://discuss.elastic.co/t/master-high-cpu/16837 "2014-04-07T07:14:52Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![RanjithVenkat](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@RanjithVenkat](https://discuss.elastic.co/u/RanjithVenkat)\ **Post date:** [April 7, 2014, 7:14am UTC](https://discuss.elastic.co/t/master-high-cpu/16837/1 "2014-04-07T07:14:52Z")
Hi, We are using Elasticsearch-0.90.5 in our search application. We have 3 master nodes and 3 data nodes each with heap size of 4gb running in JDK 1.6. We have our Node Client running in our APP\_SERVER. Elasticsearch master gets HIGH\_CPU frequently at least for every 20 days and it is not coming down until that node is restarted. When i see thread dump, nothing peculiar except GC. Number of GC threads is almost 20. _GC MarkSweep Frequency_ and _GC ParNew Frequency_ is very very less when compared to other nodes. When i restart the master, system load becomes normal. But our APP\_SERVER gets HIGH\_CPU. With our monitoring tool, we could detect some packets were dropped due to network buffer overflow. When we restart APP\_SERVER, all things become normal. For every 20 days, we have to restart our master and then our APP\_SERVERS. This has become our routine. I have attached Elasticsearch Masters thread dump here [https://gist.github.com/anonymous/10015420](https://gist.github.com/anonymous/10015420) We couldn't analyze why we are getting HIGH\_CPU repeatedly. Any help in resolving this issue will be appreciated. Thanks in advance Ranjith Venkatesan PS : Elasticsearch version and JDK version cant be updated immediately in our case. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![RanjithVenkat](https://avatars.discourse-cdn.com/v4/letter/r/c5a1d2/32.png) [@RanjithVenkat](https://discuss.elastic.co/u/RanjithVenkat)\ **Post date:** [April 8, 2014, 5:18am UTC](https://discuss.elastic.co/t/master-high-cpu/16837/2 "2014-04-08T05:18:23Z")
I was searching for some solution, i came across this one. > > > I have written a socket server for messages using netty 4.0.9-FInal. But even wh…ile handling a single connection, CPU load goes upto 100%. > > My setup: > 1. amazon standard ami (centos) > 2. java7u40 (which is supposed to have bug fix http://bugs.sun.com/view\_bug.do?bug\_id=6403933 > > What all I have tried > 1. http://code.google.com/p/spymemcached/issues/detail?id=279 > This thing didn't help. > 2. Attached Executor workers so as to keep min of load on nio worker groups. > 3. setAutoRead is set to false whenever possible. TCP\_NO\_DELAY, RECV BUFFER, SEND\_BUFFER are set to optmise performance. > 4. I tuned up some kernel parameters related with buffers to maximize performance. > > Is there anything that I am missing ? Whether my issue is related to this one ? On Mon, Apr 7, 2014 at 12:44 PM, Ranjith Venkatesan \< [ranjith.venkatesan@gmail.com](mailto:ranjith.venkatesan@gmail.com)\> wrote: > Hi, > > We are using Elasticsearch-0.90.5 in our search application. We have 3 > master nodes and 3 data nodes each with heap size of 4gb running in JDK > 1.6. We have our Node Client running in our APP\_SERVER. Elasticsearch > master gets HIGH\_CPU frequently at least for every 20 days and it is not > coming down until that node is restarted. When i see thread dump, nothing > peculiar except GC. Number of GC threads is almost 20. _GC MarkSweep > Frequency_ and _GC ParNew Frequency_ is very very less when compared to > other nodes. > > When i restart the master, system load becomes normal. But our APP\_SERVER > gets HIGH\_CPU. With our monitoring tool, we could detect some packets were > dropped due to network buffer overflow. When we restart APP\_SERVER, all > things become normal. > > For every 20 days, we have to restart our master and then our APP\_SERVERS. > This has become our routine. I have attached Elasticsearch Masters thread > dump here [https://gist.github.com/anonymous/10015420](https://gist.github.com/anonymous/10015420) > > We couldn't analyze why we are getting HIGH\_CPU repeatedly. Any help in > resolving this issue will be appreciated. > > Thanks in advance > > Ranjith Venkatesan > > PS : Elasticsearch version and JDK version cant be updated immediately in > our case. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/aglssMPhYQk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/aglssMPhYQk/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAM2Z\_7DKMSLVAaeuTVy0OhZ6Z%3DYQ%2Bs1LMdnV2\_yJty2aH51Ydw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAM2Z_7DKMSLVAaeuTVy0OhZ6Z%3DYQ%2Bs1LMdnV2_yJty2aH51Ydw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [April 8, 2014, 5:31am UTC](https://discuss.elastic.co/t/master-high-cpu/16837/3 "2014-04-08T05:31:50Z")
Sounds like your cluster is reaching capacity. I'd try adding more data nodes, if you cannot upgrade java and ES. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 8 April 2014 15:18, Ranjith [ranjith.venkatesan@gmail.com](mailto:ranjith.venkatesan@gmail.com) wrote: > I was searching for some solution, i came across this one. > > [100 % CPU usage even with oracle java7u40 · Issue #1873 · netty/netty · GitHub](https://github.com/netty/netty/issues/1873) > > Whether my issue is related to this one ? > > On Mon, Apr 7, 2014 at 12:44 PM, Ranjith Venkatesan \< > [ranjith.venkatesan@gmail.com](mailto:ranjith.venkatesan@gmail.com)\> wrote: > > > Hi, > > > > We are using Elasticsearch-0.90.5 in our search application. We have 3 > > master nodes and 3 data nodes each with heap size of 4gb running in JDK > > 1.6. We have our Node Client running in our APP\_SERVER. Elasticsearch > > master gets HIGH\_CPU frequently at least for every 20 days and it is not > > coming down until that node is restarted. When i see thread dump, nothing > > peculiar except GC. Number of GC threads is almost 20. _GC MarkSweep > > Frequency_ and _GC ParNew Frequency_ is very very less when compared to > > other nodes. > > > > When i restart the master, system load becomes normal. But our APP\_SERVER > > gets HIGH\_CPU. With our monitoring tool, we could detect some packets were > > dropped due to network buffer overflow. When we restart APP\_SERVER, all > > things become normal. > > > > For every 20 days, we have to restart our master and then our > > APP\_SERVERS. This has become our routine. I have attached Elasticsearch > > Masters thread dump here [https://gist.github.com/anonymous/10015420](https://gist.github.com/anonymous/10015420) > > > > We couldn't analyze why we are getting HIGH\_CPU repeatedly. Any help in > > resolving this issue will be appreciated. > > > > Thanks in advance > > > > Ranjith Venkatesan > > > > PS : Elasticsearch version and JDK version cant be updated immediately in > > our case. > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/aglssMPhYQk/unsubscribe](https://groups.google.com/d/topic/elasticsearch/aglssMPhYQk/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAM2Z\_7DKMSLVAaeuTVy0OhZ6Z%3DYQ%2Bs1LMdnV2\_yJty2aH51Ydw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAM2Z_7DKMSLVAaeuTVy0OhZ6Z%3DYQ%2Bs1LMdnV2_yJty2aH51Ydw%40mail.gmail.com)[https://groups.google.com/d/msgid/elasticsearch/CAM2Z\_7DKMSLVAaeuTVy0OhZ6Z%3DYQ%2Bs1LMdnV2\_yJty2aH51Ydw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAM2Z_7DKMSLVAaeuTVy0OhZ6Z%3DYQ%2Bs1LMdnV2_yJty2aH51Ydw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624ZEbNOqFLw0BwJfX\_%3Dy\_17HLQ\_VLfcShw3GLQZnc4f%3DGw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624ZEbNOqFLw0BwJfX_%3Dy_17HLQ_VLfcShw3GLQZnc4f%3DGw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:37am UTC](https://discuss.elastic.co/t/master-high-cpu/16837/4 "2017-07-06T01:37:41Z")