Error In Opening Dashboard .[[gc] overhead]

Elastic Stack Kibana
1

hi every one , there is a problem in my cluster after the data getting bigger and bigger the problem occur more , the issue is my dashboard didn't load when the data increased from 800GB to 1.5 TB my cluster is ok for every transaction but for opening this dashboard i have a error that raised in elasticsearch log like something like this ::

> [2019-07-16T11:04:10,765][INFO ][o.e.m.j.JvmGcMonitorService] [DataNode-02] [gc][72362] overhead, spent [362ms] collecting in the last [1.1s]
> [2019-07-16T11:04:36,068][INFO ][o.e.m.j.JvmGcMonitorService] [DataNode-02] [gc][72387] overhead, spent [283ms] collecting in the last [1.1s]
> [2019-07-16T11:04:47,070][INFO ][o.e.m.j.JvmGcMonitorService] [DataNode-02] [gc][72398] overhead, spent [353ms] collecting in the last [1s]
> [2019-07-16T11:04:50,071][INFO ][o.e.m.j.JvmGcMonitorService] [DataNode-02] [gc][72401] overhead, spent [332ms] collecting in the last [1s]

how can i solve this problem ?

2

What is the specification of your cluster in terms of version, node count, hardware and heap sizes? How many indices and shards do you have in the cluster?

3

Elasticsearch : Version 6.5.4
Node Count : 6
Indices : 122
Shards : 854
hardware : storage : 3 TB
Memory : 306 GB
Heap Size : 165 GB
Cpu Cores : 36 Core

4

anyone know the problem ????

5

What does that dashboard like like?

6

we have about 20 visualization in a dashboard on about 1.3 TB stored data

7

What does disk I/O, iowait and CPU usage look like when you load this dashboard?

8

about 97% in 3 servers and other as usual ,
another question that i want to ask , is my heap size ok ? the server has 64 GB Ram and i set the heap size just 32 GB , Can i set more Heap size in this server ?

9

Heap should usually not be larger than 30GB. Is that 97% disk utilisation?

10

no 97% of CPU Usage , three server going up to 95% 97% and kibana didnt respond for 4 or 5 minute

11

image

when i click the dashboard my node result is like this :

12

Can you check I/O statistics using ‘iostat’ when you are loading the dashboard? What kind of storage do you have? SSDs?

13

this is the result what you want ::

no there is no SSD Hard Drive , All Hard Disks are HDD.

thank you.

iostat1
iostat1.PNG1032×523 84.7 KB
iostat2
iostat2.png1449×709 172 KB

14

Any Idea? ? ?

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.