Message "timeout notification from cluster service"

Elastic Stack Elasticsearch
1

One of our nodes (N1) has been very frequently reporting the message "timeout notification from cluster service". I referred to ElasticSearch : observer: timeout notification from cluster service to check if that's got anything to do with gc on the other node (N2) in the cluster. The other node in the cluster doesn't have any slow gc runs, only one such gc run is logged. Where as N1 logs report about 20K such timeout messages over a windows of 24 hours.

Other than the gc run in progress, what else could a node(N1 in this case) to throw "timeout notification from cluster service" while waiting for the other node(N2)? Would it happen if the load on the other node (N2) was too high?

Please share your thoughts!

2

As seen on logs of N1

es@node1:/var/log/elasticsearch$ grep -irs "timeout notification from cluster service" es-n1.log |egrep "2017-03-21 [0-2][0-9]:"|wc -l
18479
es@node1:/var/log/elasticsearch$ grep -irs "timeout notification from cluster service" es-n1.log |egrep "2017-03-21 1[0-9]:"|wc -l
4
es@node1:/var/log/elasticsearch$

As seen on logs of N2

es@node2:/var/log/elasticsearch$ egrep "1[0-9]:[0-9][0-9]:" es-n2.log | egrep -iw gc
es@node2:/var/log/elasticsearch$

About 18K such messages seen on N1 where as logs in N2 do not hint at any trouble.

3

Are they in the same datacentre/region?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.