Hi,
my kibana is quite slow. Coming from kibana 3, it seems to be faster than the new one.
We have a single node instance, 9 CPU 24GB RAM. Running Logstash, ELK and Kibana on the same machine.
sporadically but quite often we get timeouts in kibana. Retrying the query mostly returns result, I think because of ES cache.
top shows:
top - 11:04:48 up 16 days, 23:19, 1 user, load average: 0.60, 1.63, 2.97
Tasks: 172 total, 2 running, 170 sleeping, 0 stopped, 0 zombie
%Cpu(s): 6.4 us, 0.6 sy, 0.5 ni, 92.4 id, 0.1 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 28780884 total, 171788 free, 18373960 used, 10235136 buff/cache
KiB Swap: 10484732 total, 10031116 free, 453616 used. 9805796 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1711 elastic+ 20 0 0.278t 0.016t 437456 S 54.5 59.0 3484:13 java
18601 logstash 39 19 7149564 1.036g 7684 S 3.7 3.8 349:00.59 java
17749 kibana 20 0 1337280 130332 5900 S 1.0 0.5 85:26.60 node
1 root 20 0 194732 4684 2792 S 0.0 0.0 8:43.57 systemd
In ES I found following in the log:
[2017-03-17T10:51:51,182][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][685485] overhead, spent [366ms] collecting in the last [1.1s]
[2017-03-17T10:51:58,263][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][685492] overhead, spent [393ms] collecting in the last [1s]
[2017-03-17T10:52:45,892][WARN ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][685538] overhead, spent [580ms] collecting in the last [1.1s]
[2017-03-17T10:52:47,009][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][685539] overhead, spent [330ms] collecting in the last [1.1s]
[2017-03-17T10:52:48,009][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][685540] overhead, spent [418ms] collecting in the last [1s]
[2017-03-17T10:52:58,011][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][685550] overhead, spent [271ms] collecting in the last [1s]
Also interesting, Why are some log lines INFO and some WARN?
Do I need to increase RAM and / or heap?
Heap settings:
ES: 15G
Logstash: 1GB
Did I badly configure the heap for ES or do we just need more RAM and bigger heap?
Thanks,
Andreas