Error In Opening Dashboard .[[gc] overhead]

hi every one , there is a problem in my cluster after the data getting bigger and bigger the problem occur more , the issue is my dashboard didn't load when the data increased from 800GB to 1.5 TB my cluster is ok for every transaction but for opening this dashboard i have a error that raised in elasticsearch log like something like this ::

> [2019-07-16T11:04:10,765][INFO ][o.e.m.j.JvmGcMonitorService] [DataNode-02] [gc][72362] overhead, spent [362ms] collecting in the last [1.1s]
> [2019-07-16T11:04:36,068][INFO ][o.e.m.j.JvmGcMonitorService] [DataNode-02] [gc][72387] overhead, spent [283ms] collecting in the last [1.1s]
> [2019-07-16T11:04:47,070][INFO ][o.e.m.j.JvmGcMonitorService] [DataNode-02] [gc][72398] overhead, spent [353ms] collecting in the last [1s]
> [2019-07-16T11:04:50,071][INFO ][o.e.m.j.JvmGcMonitorService] [DataNode-02] [gc][72401] overhead, spent [332ms] collecting in the last [1s]

how can i solve this problem ?

What is the specification of your cluster in terms of version, node count, hardware and heap sizes? How many indices and shards do you have in the cluster?

Elasticsearch : Version 6.5.4
Node Count : 6
Indices : 122
Shards : 854
hardware : storage : 3 TB
Memory : 306 GB
Heap Size : 165 GB
Cpu Cores : 36 Core

anyone know the problem ????

What does that dashboard like like?

we have about 20 visualization in a dashboard on about 1.3 TB stored data

What does disk I/O, iowait and CPU usage look like when you load this dashboard?

about 97% in 3 servers and other as usual ,
another question that i want to ask , is my heap size ok ? the server has 64 GB Ram and i set the heap size just 32 GB , Can i set more Heap size in this server ?

Heap should usually not be larger than 30GB. Is that 97% disk utilisation?

no 97% of CPU Usage , three server going up to 95% 97% and kibana didnt respond for 4 or 5 minute


when i click the dashboard my node result is like this :

Can you check I/O statistics using ‘iostat’ when you are loading the dashboard? What kind of storage do you have? SSDs?

this is the result what you want ::

no there is no SSD Hard Drive , All Hard Disks are HDD.

thank you.

Any Idea? ? ?