Elasticsearch consumes all CPU

Elastic Stack Elasticsearch
1

Hello all,

ES became unresponsive. I checked the log and it's full of lines like:

[2012-09-25 22:52:02,433][WARN ][monitor.jvm ]
[Destroyer] [gc][ParNew][133651][2563] duration [4.3s], collections
[1]/[4.4s], total [4.3s]/[40.4m], memory [6.6gb]->[6.7gb]/[8.9gb],
all_pools {[Code Cache] [8.3mb]->[8.3mb]/[48mb]}{[Par Eden Space]
[1.4mb]->[3.5mb]/[66.5mb]}{[Par Survivor Space] [8.3mb]->[8.3mb]/
[8.3mb]}{[CMS Old Gen] [6.6gb]->[6.7gb]/[8.9gb]}{[CMS Perm Gen] [38mb]-

[38mb]/[82mb]}

According to the log, there is still plenty of RAM left, but CPU is
stuck at full 400% (4 cores) for the java process. Any idea what's
going on?

This is a 13GB VM, with 9GB dedicated to JVM (Sun 20.1-b02) with
mlockall, 4 cores, no load.

I tried inspecting with bigdesk, but bigdesk won't even load up, as
all cores are so busy... Restarting the machine doesn't help either --
ES starts normally, but the first query that comes in spins up CPU to
400% again. I am a bit desperate about how to debug this.

--

2

Hello,

Use SPM for Elasticsearch (see sig), it will help you out.
To have a closer look at what the JVM is going you can also use jstat
-gcutil PID 2000 100

Otis

Search Analytics - Cloud Monitoring Tools & Services | Sematext
Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Tuesday, September 25, 2012 5:12:12 PM UTC-4, Crwe wrote:

Hello all,

ES became unresponsive. I checked the log and it's full of lines like:

[2012-09-25 22:52:02,433][WARN ][monitor.jvm ]
[Destroyer] [gc][ParNew][133651][2563] duration [4.3s], collections
[1]/[4.4s], total [4.3s]/[40.4m], memory [6.6gb]->[6.7gb]/[8.9gb],
all_pools {[Code Cache] [8.3mb]->[8.3mb]/[48mb]}{[Par Eden Space]
[1.4mb]->[3.5mb]/[66.5mb]}{[Par Survivor Space] [8.3mb]->[8.3mb]/
[8.3mb]}{[CMS Old Gen] [6.6gb]->[6.7gb]/[8.9gb]}{[CMS Perm Gen] [38mb]-

[38mb]/[82mb]}

According to the log, there is still plenty of RAM left, but CPU is
stuck at full 400% (4 cores) for the java process. Any idea what's
going on?

This is a 13GB VM, with 9GB dedicated to JVM (Sun 20.1-b02) with
mlockall, 4 cores, no load.

I tried inspecting with bigdesk, but bigdesk won't even load up, as
all cores are so busy... Restarting the machine doesn't help either --
ES starts normally, but the first query that comes in spins up CPU to
400% again. I am a bit desperate about how to debug this.

--