Elasticsearch getting stucked after few iterations

Elastic Stack Elasticsearch
1

Hello,

I am using ES ver 1.7.4 and we are using java api to search and make connection to es. Searching and making connection is in parallel, we have observed that after few minutes elasticsearch getting stucked, neither we are getting any new connection and the search queries are all stuck. After 1 minutes all the porcess resumes and then the operations work normally. This is impacting the performance of our application.

We have 5 nodes in cluster with 32 Gb Ram and 16 cores processor, one es index with 5 shards no replica, no of documents is approx 480 million.

Please help.

Thanks,
Prashant

2

Can you share thread dumps of application when it is in hang state.
Please check the ES GC behaviour? is there any long GC pauses?
Also, u can use hot threads rest API to find what ES cluster is doing during the hang state.
use below:
http://localhost:9200/_nodes/hotthreads

3

Did not you ask a similar question here?

Probably you have Long GC running? Do you see anything in logs?

4

No there is nothing in the log,...will share the thread dump

5

GC Log...

[2016-06-29 10:32:35,782][INFO ][monitor.jvm ] [match 3] [gc][young][1704][10] duration [876ms], collections [1]/[1.5s], total [876ms]/[4.9s], memory [3.6gb]->[3.5gb]/[9.8gb], all_pools {[young] [830.5mb]->[4mb]/[865.3mb]}{[survivor] [108.1mb]->[108.1mb]/[108.1mb]}{[old] [2.7gb]->[3.4gb]/[8.9gb]}