I have an elasticsearch cluster (v1.7.5) on Ubuntu (4 data, 3 masters).
Normal operation looks fine - nice response time.
This is the case when used behind a load balancer, or when I stipulate all node IPs in my cluster and the client (my application) connects directly to each node.
However, intermittently, elasticsearch does not accept new connections. By this I mean, the TCP 3 way handshake is not completed; the initial SYN frame arrives at the host, on the correct port, but no SYN, ACK frame is sent back.
I have marvel, and looked at the request queues (LISTENER THREAD POOL REJECTED), and they seem fine (0 rejections).
I cannot see anything in the logs that ever said anything to do with network issues, and I was wondering if
a) Anyone has similar issues
b) Anyone can point me in a direction for debugging further
Thanks for your assistance in advance!