One node frequently goes into 100% CPU and GC loop


(Omar Al Zabir) #1

ES 2.1.0. I have 3 node cluster, identical config. One node always goes into 100% CPU and continuous GC when I send high traffic. Even if I stop sending any traffic, that node always remains in 100% CPU and continuous GC loop, while the other 2 nodes are back to 0% CPU.
Here's log from defective node:

[2016-01-23 12:41:54,568][INFO ][monitor.jvm              ] [ec-dyl09026app04] [gc][old][77636][706] duration [5.8s], collections [1]/[6.2s], total [5.8s]/[2.6m], memory [972.4mb]->[976.5mb]/[990.7mb], all_pools {[young] [266.2mb]->[266.2mb]/[266.2mb]}{[survivor] [14.9mb]->[19mb]/[33.2mb]}{[old] [691.2mb]->[691.2mb]/[691.2mb]}
[2016-01-23 12:42:03,107][INFO ][monitor.jvm              ] [ec-dyl09026app04] [gc][old][77637][707] duration [8.2s], collections [1]/[8.5s], total [8.2s]/[2.7m], memory [976.5mb]->[980.2mb]/[990.7mb], all_pools {[young] [266.2mb]->[266.2mb]/[266.2mb]}{[survivor] [19mb]->[22.7mb]/[33.2mb]}{[old] [691.2mb]->[691.2mb]/[691.2mb]}
[2016-01-23 12:42:09,816][INFO ][monitor.jvm              ] [ec-dyl09026app04] [gc][old][77638][708] duration [6.3s], collections [1]/[6.6s], total [6.3s]/[2.8m], memory [980.2mb]->[986.5mb]/[990.7mb], all_pools {[young] [266.2mb]->[266.2mb]/[266.2mb]}{[survivor] [22.7mb]->[29.8mb]/[33.2mb]}{[old] [691.2mb]->[691.2mb]/[691.2mb]}
[2016-01-23 12:42:24,085][INFO ][monitor.jvm              ] [ec-dyl09026app04] [gc][old][77639][710] duration [14.1s], collections [2]/[14.2s], total [14.1s]/[3.1m], memory [986.5mb]->[990.5mb]/[990.7mb], all_pools {[young] [266.2mb]->[266.2mb]/[266.2mb]}{[survivor] [29.8mb]->[33mb]/[33.2mb]}{[old] [691.2mb]->[691.2mb]/[691.2mb]}

(Mark Walkom) #2

You don't seem to have much heap for these nodes.

How much data are you sending and how much is in the cluster?


(Christian Dahlqvist) #3

Are you distributing requests evenly across the 3 nodes?


(system) #4