We have a 11 node cluster[3master, 3query, 5data]. Two nodes [Master1 and Query1] suddenly left the cluster for around 30 mins .
All I could see in the logs is
Caught exception while handling client http traffic, closing connection...
java.io.IOException: An existing connection was forcibly closed by the remote host
connect_timeout[30s]]; nested: ConnectException[Connection timed out: no further information: ];
The ES dashboards indicates that the memory/processortime/heap usage was very normal and there weren't any spikes.
Other than the fact that Master1 and Query1 were re-started on ES crashing, how do i verify why it crashed in the first place ?
Any help or clarification here would be really appreciated. Thanks in advance!