SIGTERM on master nodes

[tetiana]

Hello, during the ES upgrade from version 7.2.1 to 7.3.2, I got:

{
  "root_cause" : [ {
    "type" : "master_not_discovered_exception",
    "reason" : null
  } ],
  "type" : "master_not_discovered_exception",
  "reason" : null
}

for each master reelection (3 master nodes)

From the logs, I see that the master election was started after the active master stopped by both left master nodes in about the same time.

Is it expected behavior that SIGTERM to an active master do not waits for a new master to be re-elected before the Elasticsearch process exits ? What is the recommended way to handle that?

[warkolm (Mark Walkom)]

SIGTERM means that the process was requested to be stopped by something. Elasticsearch cannot do that by itself.

[Christian_Dahlqvist]

In a cluster it is the master leaving that triggers election. This can not be initiated before the current master leaves.

[tetiana]

Thank you for your replies.

Somehow during the upgrade, one of reelections took 2m33s. What could be recommended way to handle that behaviour? Add one more master node (so to get 4 in total, seems it should be handled correctly: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-voting.html#_even_numbers_of_master_eligible_nodes)? any other ideas?