One of our nodes (N1) has been very frequently reporting the message "timeout notification from cluster service". I referred to ElasticSearch : observer: timeout notification from cluster service to check if that's got anything to do with gc on the other node (N2) in the cluster. The other node in the cluster doesn't have any slow gc runs, only one such gc run is logged. Where as N1 logs report about 20K such timeout messages over a windows of 24 hours.
Other than the gc run in progress, what else could a node(N1 in this case) to throw "timeout notification from cluster service" while waiting for the other node(N2)? Would it happen if the load on the other node (N2) was too high?
Please share your thoughts!