We are using ElasticSearch(2.1.1) , with Logstash 2.1.1 and Kibana 4.3.1 in a production environment of a national payment switch in India. Unfortunately after running smoothly for 6 months, ElasticSearch started hanging a few times and coming up automatically within 15 minutes. The total size of all index files is around 1.2 GB. The error is as below. Any solution is welcome.
[2016-10-07 10:11:23,812][DEBUG][action.admin.cluster.node.stats] [pr_app2] failed to execute on node [-EEHqL06RteY7ROq4l4Bhg] ReceiveTimeoutTransportException[[pr_app2][172.16.173.12:9301][cluster:monitor/nodes/stats[n]] request_id [360034727] timed out after [16011ms]]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:645)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-10-07 10:12:02,742][WARN ][transport ] [pr_app2] Received response for a request that has timed out, sent [54943ms] ago, timed out [38932ms] ago, action [cluster:monitor/nodes/stats[n]], node [{pr_app2}{-EEHqL06RteY7ROq4l4Bhg}{172.16.173.12}{172.16.173.12:9301}{master=true}], id [360034727]