Elasticsearch, queries and data transfer are very slow and I am getting timeout errors. I found this information in the log files of the data nodes.
Elasticsearch and Kibana version 7.4.2
I have 13 nodes (3 master ,10 data nodes)
Each node has 64 GB of RAM and 5 TB of storage.
Each node jvm.options -Xms30g -Xmx30g
[2021-01-06T09:38:30,253][INFO ][o.e.m.j.JvmGcMonitorService] [data-5] [gc][31546102] overhead, spent [287ms] collecting in the last [1s]
[2021-01-06T09:55:12,720][INFO ][o.e.m.j.JvmGcMonitorService] [data-5] [gc][31547354] overhead, spent [303ms] collecting in the last [1s]
[2021-01-06T10:04:17,502][INFO ][o.e.m.j.JvmGcMonitorService] [data-5] [gc][31546102] overhead, spent [287ms] collecting in the last [1.2s]
[2021-01-06T10:54:54,473][INFO ][o.e.m.j.JvmGcMonitorService] [data-5] [gc][31552295] overhead, spent [340ms] collecting in the last [1s]
[2021-01-06T11:41:58,347][WARN ][o.e.m.j.JvmGcMonitorService] [data-5] [gc][31555117] overhead, spent [521ms] collecting in the last [1s]
GET _cluster / stats çıktısı