I have an ES cluster with 27 nodes (3 master, 24 data). At times I see a
burst of nodes leaving and rejoining within couple of minutes. Each node
has 16GB allocated for the JVM heap and are not close to touching those
limits. There are no memory issues, and there is no search/index operations
going on when this occurred. But there are quite a few nodedisconnected
messages that suddenly appear on the master. It doesn’t seem to happen all
the time but in bursts.
During this time, on the master, I see NodeDisconnectedException for a
node. On that node, I see messages that say “master left (reason =
transport disconnected)”. I don't think its split-brain though with the
number of messages in the logs its hard to figure out. Also min number of
master setting is set to 2. The outcome is that it causes a whole lot of
shards to shift around.
I'd like to involve our network specialists to troubleshoot
connectivity but not sure what to ask them to look for. In what scenarios
does ElasticSearch reports node disconnected? Should they be looking at TCP
connectivity, run some ping tests, etc.?
Also are there timeout values that can be configured so we can reduce false
positives for node disconnected events?
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to email@example.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e7ad5de3-0e9b-4496-9c96-5162b784bac1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.