Recently brought up a 2 node cluster running 1.5.2-1 and things were working fine. Its been about 2 weeks now and for the past few days I have had to restart the nodes for the cluster to recover and become useable again.
I am running 2 r3.xlarge instances within AWS each configured with a 1TB GP2 EBS volume and using the ec2 discovery method. When things start to go bad the last thing reported in the log is
Caused by: org.elasticsearch.transport.TransportException: TransportService is closed stopped can't send request
When I look at the bigdesk plugin cluster health shows green however all I get from Kibana is a Gateway error when trying a search. If I restart elasticsearch things seem to be fine for around 20 hours or so and then the same situation reoccurs.
I also have a single node cluster that I used as a test machine and it has never displayed this behavior.
Can I provide further details/configuration to help diagnose if I have misconfigured in some way. Or help determine if I am hitting some kind of resource issue?