I'm getting about 600-1000% CPU usage on a 12 core machine running logstash @6000 eps. Although I wonder what is this 100% usage on 1 cpu with 25% user and 75% system. Seems to be related to all the java thread running. Is this normal? Is there a way to distribute this load across other CPUs?
top - 20:06:19 up 45 days, 14 min, 1 user, load average: 1.32, 1.88, 1.74
Tasks: 18 total, 1 running, 17 sleeping, 0 stopped, 0 zombie
%Cpu0 : 0.0 us, 0.7 sy, 18.0 ni, 81.3 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu1 : 24.9 us, 75.1 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu2 : 0.0 us, 0.3 sy, 10.8 ni, 88.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu3 : 0.0 us, 0.0 sy, 19.0 ni, 81.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu4 : 0.0 us, 0.3 sy, 7.7 ni, 91.9 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu5 : 0.0 us, 0.0 sy, 12.0 ni, 88.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
%Cpu6 : 0.0 us, 0.7 sy, 16.3 ni, 82.7 id, 0.0 wa, 0.0 hi, 0.3 si, 0.0 st
...
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
35151 logstash 39 19 11.971g 815080 14716 S 20.6 2.5 1:33.69 java
35190 logstash 39 19 11.971g 815080 14716 S 13.3 2.5 0:36.79 [main]>worker0
35208 logstash 39 19 11.971g 815080 14716 S 13.3 2.5 0:39.11 [main]>worker18
35198 logstash 39 19 11.971g 815080 14716 S 12.6 2.5 0:35.19 [main]>worker8
35200 logstash 39 19 11.971g 815080 14716 S 11.6 2.5 0:40.60 [main]>worker10
35211 logstash 39 19 11.971g 815080 14716 S 11.6 2.5 0:40.26 [main]>worker21
35196 logstash 39 19 11.971g 815080 14716 S 10.6 2.5 0:40.05 [main]>worker6
35204 logstash 39 19 11.971g 815080 14716 S 10.6 2.5 0:34.66 [main]>worker14
35202 logstash 39 19 11.971g 815080 14716 S 10.3 2.5 0:38.24 [main]>worker12
35210 logstash 39 19 11.971g 815080 14716 S 10.3 2.5 0:34.93 [main]>worker20
35194 logstash 39 19 11.971g 815080 14716 S 9.6 2.5 0:39.67 [main]>worker4
35206 logstash 39 19 11.971g 815080 14716 S 9.6 2.5 0:42.55 [main]>worker16
35193 logstash 39 19 11.971g 815080 14716 S 8.6 2.5 0:39.77 [main]>worker3
35195 logstash 39 19 11.971g 815080 14716 S 8.6 2.5 0:35.13 [main]>worker5
35239 logstash 39 19 11.971g 815080 14716 S 8.6 2.5 0:17.17 [beats-input]>c
35289 logstash 39 19 11.971g 815080 14716 S 8.6 2.5 0:18.11 [beats-input]>c
35191 logstash 39 19 11.971g 815080 14716 S 8.3 2.5 0:39.72 [main]>worker1
35197 logstash 39 19 11.971g 815080 14716 S 8.3 2.5 0:42.74 [main]>worker7
35212 logstash 39 19 11.971g 815080 14716 S 8.3 2.5 0:38.53 [main]>worker22
35283 logstash 39 19 11.971g 815080 14716 S 8.3 2.5 0:15.41 [beats-input]>c
35240 logstash 39 19 11.971g 815080 14716 S 8.0 2.5 0:17.21 [beats-input]>c
35192 logstash 39 19 11.971g 815080 14716 S 7.6 2.5 0:44.77 [main]>worker2
35199 logstash 39 19 11.971g 815080 14716 S 7.6 2.5 0:38.76 [main]>worker9
35205 logstash 39 19 11.971g 815080 14716 S 7.6 2.5 0:36.82 [main]>worker15
35209 logstash 39 19 11.971g 815080 14716 S 7.6 2.5 0:34.97 [main]>worker19
35213 logstash 39 19 11.971g 815080 14716 S 7.6 2.5 0:42.10 [main]>worker23
35128 logstash 39 19 11.971g 815080 14716 S 6.0 2.5 0:22.78 java
35132 logstash 39 19 11.971g 815080 14716 S 6.0 2.5 0:22.74 java
35145 logstash 39 19 11.971g 815080 14716 S 6.0 2.5 0:22.68 java
35130 logstash 39 19 11.971g 815080 14716 S 5.6 2.5 0:22.66 java
35131 logstash 39 19 11.971g 815080 14716 S 5.6 2.5 0:22.74 java
35133 logstash 39 19 11.971g 815080 14716 S 5.6 2.5 0:22.73 java
35136 logstash 39 19 11.971g 815080 14716 S 5.6 2.5 0:22.77 java
35139 logstash 39 19 11.971g 815080 14716 S 5.6 2.5 0:22.75 java
35140 logstash 39 19 11.971g 815080 14716 S 5.6 2.5 0:22.71 java
35143 logstash 39 19 11.971g 815080 14716 S 5.6 2.5 0:22.74 java
35144 logstash 39 19 11.971g 815080 14716 S 5.6 2.5 0:22.68 java
... much more...