Request Timeout after 30000ms for kibana

[Amin1 (Amin)]

hi all
I have 2 client node - 3 data node - 2 master node - kibana that connected to Client node.
after one query that sends by kibana, it take more time and after that i get the following error in kibana :

:

" plugin:elasticsearch@5.4.0 Request Timeout after 30000ms "

after that my nodes dont response to the following command :
" curl -XGET 'http://172.24.69.17:9200/_cat/nodes' "

[warkolm (Mark Walkom)]

What do your ES logs show?

[Amin1 (Amin)]

how I can see or check logs ?
and i check log from wick node ?

[warkolm (Mark Walkom)]

Login to the Elasticsearch host above and look in /var/log/elasticsearch/.

[Amin1 (Amin)]

This is log Of one the Data Nodes :
[root@es-data-02 ~]# tail -F /var/log/elasticsearch/logserver-2017-07-03.log
... 40 more
[2017-07-03T23:59:22,059][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-02] [gc][4154598] overhead, spent [4.7s] collecting in the last [4.9s]
[2017-07-03T23:59:29,319][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-02] [gc][old][4154600][25106] duration [6s], collections [1]/[6.2s], total [6s]/[16.7h], memory [1.8gb]->[1.7gb]/[1.9gb], all_pools {[young] [218.7mb]->[53.7mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] [1.6gb]->[1.6gb]/[1.6gb]}
[2017-07-03T23:59:29,319][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-02] [gc][4154600] overhead, spent [6s] collecting in the last [6.2s]
[2017-07-03T23:59:35,076][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-02] [gc][4154601] overhead, spent [4.9s] collecting in the last [5.7s]
[2017-07-03T23:59:42,876][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-02] [gc][old][4154602][25108] duration [7.2s], collections [1]/[7.7s], total [7.2s]/[16.7h], memory [1.7gb]->[1.7gb]/[1.9gb], all_pools {[young] [64.7mb]->[51.1mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] [1.6gb]->[1.6gb]/[1.6gb]}
[2017-07-03T23:59:42,876][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-02] [gc][4154602] overhead, spent [7.2s] collecting in the last [7.7s]
[2017-07-03T23:59:48,937][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-02] [gc][4154604] overhead, spent [4.8s] collecting in the last [5s]
[2017-07-03T23:59:56,597][INFO ][o.e.m.j.JvmGcMonitorService] [es-data-02] [gc][old][4154606][25110] duration [6.3s], collections [1]/[6.6s], total [6.3s]/[16.8h], memory [1.8gb]->[1.7gb]/[1.9gb], all_pools {[young] [228.4mb]->[71mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] [1.6gb]->[1.6gb]/[1.6gb]}
[2017-07-03T23:59:56,597][WARN ][o.e.m.j.JvmGcMonitorService] [es-data-02] [gc][4154606] overhead, spent [6.3s] collecting in the last [6.6s]

[warkolm (Mark Walkom)]

That's very high garbage collection.

How large is your heap?

[Amin1 (Amin)]

Hi
I delete some shards and problem solvedI appreciate your attention and time
best