Master High CPU


(RanjithVenkat) #1

Hi,

We are using Elasticsearch-0.90.5 in our search application. We have 3
master nodes and 3 data nodes each with heap size of 4gb running in JDK
1.6. We have our Node Client running in our APP_SERVER. Elasticsearch
master gets HIGH_CPU frequently at least for every 20 days and it is not
coming down until that node is restarted. When i see thread dump, nothing
peculiar except GC. Number of GC threads is almost 20. GC MarkSweep
Frequency
and GC ParNew Frequency is very very less when compared to
other nodes.

When i restart the master, system load becomes normal. But our APP_SERVER
gets HIGH_CPU. With our monitoring tool, we could detect some packets were
dropped due to network buffer overflow. When we restart APP_SERVER, all
things become normal.

For every 20 days, we have to restart our master and then our APP_SERVERS.
This has become our routine. I have attached Elasticsearch Masters thread
dump here https://gist.github.com/anonymous/10015420

We couldn't analyze why we are getting HIGH_CPU repeatedly. Any help in
resolving this issue will be appreciated.

Thanks in advance

Ranjith Venkatesan

PS : Elasticsearch version and JDK version cant be updated immediately in
our case.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(RanjithVenkat) #2

I was searching for some solution, i came across this one.

Whether my issue is related to this one ?

On Mon, Apr 7, 2014 at 12:44 PM, Ranjith Venkatesan <
ranjith.venkatesan@gmail.com> wrote:

Hi,

We are using Elasticsearch-0.90.5 in our search application. We have 3
master nodes and 3 data nodes each with heap size of 4gb running in JDK
1.6. We have our Node Client running in our APP_SERVER. Elasticsearch
master gets HIGH_CPU frequently at least for every 20 days and it is not
coming down until that node is restarted. When i see thread dump, nothing
peculiar except GC. Number of GC threads is almost 20. GC MarkSweep
Frequency
and GC ParNew Frequency is very very less when compared to
other nodes.

When i restart the master, system load becomes normal. But our APP_SERVER
gets HIGH_CPU. With our monitoring tool, we could detect some packets were
dropped due to network buffer overflow. When we restart APP_SERVER, all
things become normal.

For every 20 days, we have to restart our master and then our APP_SERVERS.
This has become our routine. I have attached Elasticsearch Masters thread
dump here https://gist.github.com/anonymous/10015420

We couldn't analyze why we are getting HIGH_CPU repeatedly. Any help in
resolving this issue will be appreciated.

Thanks in advance

Ranjith Venkatesan

PS : Elasticsearch version and JDK version cant be updated immediately in
our case.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/aglssMPhYQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAM2Z_7DKMSLVAaeuTVy0OhZ6Z%3DYQ%2Bs1LMdnV2_yJty2aH51Ydw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #3

Sounds like your cluster is reaching capacity. I'd try adding more data
nodes, if you cannot upgrade java and ES.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 8 April 2014 15:18, Ranjith ranjith.venkatesan@gmail.com wrote:

I was searching for some solution, i came across this one.

https://github.com/netty/netty/issues/1873

Whether my issue is related to this one ?

On Mon, Apr 7, 2014 at 12:44 PM, Ranjith Venkatesan <
ranjith.venkatesan@gmail.com> wrote:

Hi,

We are using Elasticsearch-0.90.5 in our search application. We have 3
master nodes and 3 data nodes each with heap size of 4gb running in JDK
1.6. We have our Node Client running in our APP_SERVER. Elasticsearch
master gets HIGH_CPU frequently at least for every 20 days and it is not
coming down until that node is restarted. When i see thread dump, nothing
peculiar except GC. Number of GC threads is almost 20. GC MarkSweep
Frequency
and GC ParNew Frequency is very very less when compared to
other nodes.

When i restart the master, system load becomes normal. But our APP_SERVER
gets HIGH_CPU. With our monitoring tool, we could detect some packets were
dropped due to network buffer overflow. When we restart APP_SERVER, all
things become normal.

For every 20 days, we have to restart our master and then our
APP_SERVERS. This has become our routine. I have attached Elasticsearch
Masters thread dump here https://gist.github.com/anonymous/10015420

We couldn't analyze why we are getting HIGH_CPU repeatedly. Any help in
resolving this issue will be appreciated.

Thanks in advance

Ranjith Venkatesan

PS : Elasticsearch version and JDK version cant be updated immediately in
our case.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/aglssMPhYQk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/aaa14b57-3253-42dc-9a67-3378bb892a3f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAM2Z_7DKMSLVAaeuTVy0OhZ6Z%3DYQ%2Bs1LMdnV2_yJty2aH51Ydw%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAM2Z_7DKMSLVAaeuTVy0OhZ6Z%3DYQ%2Bs1LMdnV2_yJty2aH51Ydw%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZEbNOqFLw0BwJfX_%3Dy_17HLQ_VLfcShw3GLQZnc4f%3DGw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4