# Master not discovered or elected yet, when stopping Elasticsearch service

**URL:** <https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-monitoring\
**Created:** [February 15, 2024, 2:13pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377 "2024-02-15T14:13:07Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![andrejze](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@andrejze](https://discuss.elastic.co/u/andrejze)\
**Post date:** [February 15, 2024, 2:13pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/1 "2024-02-15T14:13:07Z")

</div>

OS: Debian 10 (buster), 4.19.0-25-amd64 #1 SMP Debian 4.19.289-2 (2023-08-08)  
Elasticsearch: 8.12.0  
Cluster: 3 nodes, all master eligible

When stopping Elasticsearch service on any node, operation gets stuck with **error :**

```auto
[2024-02-15T14:45:21,656][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html

```

It continues to show this error, until the java process is killed with SIGKILL signal.

**My elasticsearch.yml:**

```auto
cluster.name: bpl-log-cluster
node.name: bpl-log-es3.my.localdomain.org
# cluster.initial_master_nodes: ["bpl-log-es1.my.localdomain.org", "bpl-log-es2.my.localdomain.org", "bpl-log-es3.my.localdomain.org"]
discovery.seed_hosts: ["bpl-log-es1.my.localdomain.org", "bpl-log-es2.my.localdomain.org", "bpl-log-es3.my.localdomain.org"]

network.host: 0.0.0.0

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/bpl-log-es_chain-nopass.pfx
xpack.security.http.ssl.truststore.path: "/etc/elasticsearch/cert_ca.jks"

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

```

---

<div class="post-metadata">

**Author:** ![andrejze](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@andrejze](https://discuss.elastic.co/u/andrejze)\
**Post date:** [February 15, 2024, 2:14pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/2 "2024-02-15T14:14:57Z")

</div>

**Elasticsearch cluster log:**

```auto
[2024-02-15T14:45:11,232][INFO][o.e.n.Node] [bpl-log-es3.my.localdomain.org] stopping ...
[2024-02-15T14:45:11,233][INFO][o.e.x.w.WatcherService] [bpl-log-es3.my.localdomain.org] stopping watch service, reason [shutdown initiated]
[2024-02-15T14:45:11,233][INFO][o.e.x.m.p.l.CppLogMessageHandler] [bpl-log-es3.my.localdomain.org] [controller/28297] [Main.cc@176] ML controller exiting
[2024-02-15T14:45:11,234][INFO][o.e.x.w.WatcherLifeCycleService] [bpl-log-es3.my.localdomain.org] watcher has stopped and shutdown
[2024-02-15T14:45:11,235][INFO][o.e.x.m.p.NativeController] [bpl-log-es3.my.localdomain.org] Native controller process has stopped - no new native processes can be started
[2024-02-15T14:45:11,651][INFO][o.e.c.c.Coordinator] [bpl-log-es3.my.localdomain.org] master node [{bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] disconnected, restarting discovery
[2024-02-15T14:45:13,681][INFO][o.e.n.Node] [bpl-log-es3.my.localdomain.org] stopped
[2024-02-15T14:45:13,681][INFO][o.e.n.Node] [bpl-log-es3.my.localdomain.org] closing ...
[2024-02-15T14:45:21,656][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:45:31,658][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:45:41,660][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:45:51,662][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:01,663][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:11,665][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:21,667][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:31,669][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:41,671][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:51,673][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:01,674][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:11,676][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:21,678][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:31,679][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:41,681][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:51,683][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:48:01,685][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:48:11,686][WARN][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:48:12,793][WARN][o.e.d.PeerFinder] [bpl-log-es3.my.localdomain.org] address [10.122.16.192:9300], node [null], requesting [false] discovery result: [][10.122.16.192:9300] connection manager is closed
[2024-02-15T14:48:12,794][WARN][o.e.d.PeerFinder] [bpl-log-es3.my.localdomain.org] address [10.122.16.193:9300], node [null], requesting [false] discovery result: [][10.122.16.193:9300] connection manager is closed
[2024-02-15T14:48:13,795][WARN][o.e.d.PeerFinder] [bpl-log-es3.my.localdomain.org] address [10.122.16.192:9300], node [null], requesting [false] discovery result: [][10.122.16.192:9300] connection manager is closed
[2024-02-15T14:48:13,796][WARN][o.e.d.PeerFinder] [bpl-log-es3.my.localdomain.org] address [10.122.16.193:9300], node [null], requesting [false] discovery result: [][10.122.16.193:9300] connection manager is closed

```

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [February 15, 2024, 2:23pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/3 "2024-02-15T14:23:09Z")

</div>

What do you have in the other nodes logs?

If I understood correctly you stopped the _es1_ node and these logs are from the _es3_ nodes, it is saying that it cannot form a quorum.

What do you have in the _es2_ node logs?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 15, 2024, 2:37pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/4 "2024-02-15T14:37:18Z")

</div>

These logs indicate you have stopped `bpl-log-es1` _and_ `bpl-log-es2` so it's expected that the cluster cannot form.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 15, 2024, 2:39pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/5 "2024-02-15T14:39:23Z")

</div>

Oh wait sorry that's not right, you're trying to shut down `bpl-log-es3`. I think you probably just need to wait for longer. If it's still going after an hour or so, we might need to look at a heap dump to understand what is stuck.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 15, 2024, 2:44pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/6 "2024-02-15T14:44:54Z")

</div>

I opened [`ClusterFormationFailureHandler` shouldn't log anything once the node is stopping · Issue #105559 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/105559) because this is a logging bug. For now, you can ignore these warning logs.

---

<div class="post-metadata">

**Author:** ![andrejze](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@andrejze](https://discuss.elastic.co/u/andrejze)\
**Post date:** [February 15, 2024, 2:48pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/7 "2024-02-15T14:48:45Z")

</div>

In the above scenario I have stopped the service on ES3 node and logs are from ES3 node. Master is ES1:

```auto
name version node.role diskUsedPercent master nodeRole nodeId http
bpl-log-es2.my.localdomain.org 8.12.0 cdfhilmrstw 69.77 - cdfhilmrstw ykzajm2GRSe3oJKNHgk8Hg 10.122.16.193:9200
bpl-log-es1.my.localdomain.org 8.12.0 cdfhilmrstw 57.87 * cdfhilmrstw sjk7jww6R7SN8I2t3oRZkQ 10.122.16.192:9200
bpl-log-es3.my.localdomain.org 8.12.0 cdfhilmrstw 32.17 - cdfhilmrstw EeCGWKRUSUaAsoaf4BZSrQ 10.122.16.194:9200

```

Both ES1 and ES2 were running and where accessible on port 9200 and 9300 from ES3 tested with nc and telnet.

Here is log from ES2 from the same time:

```auto
[2024-02-15T14:45:11,652][INFO][o.e.t.ClusterConnectionManager] [bpl-log-es2.my.localdomain.org] transport connection to [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] closed by remote

[2024-02-15T14:45:11,835][INFO][o.e.c.s.ClusterApplierService] [bpl-log-es2.my.localdomain.org] removed {{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}}, term: 244, version: 262058, reason: ApplyCommitRequest{term=244, version=262058, sourceNode={bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}{ml.machine_memory=33731747840, xpack.installed=true, transform.config_version=10.0.0, ml.config_version=12.0.0, ml.max_jvm_size=17179869184, ml.allocated_processors_double=8.0, ml.allocated_processors=8}}

[2024-02-15T14:45:11,867][INFO][o.e.i.s.IndexShard] [bpl-log-es2.my.localdomain.org] [.tasks][0] primary-replica resync completed with 0 operations

[2024-02-15T14:45:11,867][INFO][o.e.i.s.IndexShard] [bpl-log-es2.my.localdomain.org] [.kibana_task_manager_7.15.0_001][0] primary-replica resync completed with 0 operations

[2024-02-15T14:47:20,337][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.memory-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,339][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.network-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,339][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.node-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,340][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.uptime-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,341][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.socket_summary-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,341][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.process.summary-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,343][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-iis.application_pool-default-2024.02.13-000015] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,343][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.cpu-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,344][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.node_stats-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,346][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.diskio-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,347][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elastic_agent.fleet_server-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,348][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.ml_job-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,348][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index-default-2024.02.13-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,349][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.ingest_pipeline-default-2024.02.13-000025] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,350][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elastic_agent.filebeat-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,351][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.process-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,351][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-iis.website-default-2024.02.09-000013] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,352][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-iis.webserver-default-2024.02.13-000015] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

```

Logs from ES1:

```auto
[2024-02-15T14:42:31,060][INFO][o.e.c.m.MetadataMappingService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.task_manager_metrics-default-2024.02.13-000016/-K-xDhV8QraPqQvpoZpjjA] update_mapping [_doc]
[2024-02-15T14:45:11,654][INFO][o.e.t.ClusterConnectionManager] [bpl-log-es1.my.localdomain.org] transport connection to [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] closed by remote
[2024-02-15T14:45:11,660][INFO][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] current.health="YELLOW" message="Cluster health status changed from [GREEN] to [YELLOW] (reason: [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008} reason: disconnected])." previous.health="GREEN" reason="{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008} reason: disconnected"
[2024-02-15T14:45:11,662][INFO][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] updating number_of_replicas to [1] for indices [.apm-source-map]
[2024-02-15T14:45:11,762][INFO][o.e.c.s.MasterService] [bpl-log-es1.my.localdomain.org] node-left[{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008} reason: disconnected], term: 244, version: 262058, delta: removed {{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}}
[2024-02-15T14:45:11,903][INFO][o.e.c.s.ClusterApplierService] [bpl-log-es1.my.localdomain.org] removed {{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}}, term: 244, version: 262058, reason: Publication{term=244, version=262058}
[2024-02-15T14:45:11,971][INFO][o.e.c.r.DelayedAllocationService] [bpl-log-es1.my.localdomain.org] scheduling reroute for delayed shards in [59.6s] (277 delayed shards)
[2024-02-15T14:45:11,971][INFO][o.e.c.c.NodeLeftExecutor] [bpl-log-es1.my.localdomain.org] node-left: [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] with reason [disconnected]
[2024-02-15T14:45:11,976][WARN][o.e.a.b.TransportShardBulkAction] [bpl-log-es1.my.localdomain.org] [[.ds-logs-system.security-default-2024.02.08-000009][0]] failed to perform indices:data/write/bulk[s] on replica [.ds-logs-system.security-default-2024.02.08-000009][0], node[EeCGWKRUSUaAsoaf4BZSrQ], [R], s[STARTED], a[id=pUYtfww_TiuqXgvXOjhOrw], failed_attempts[0]
org.elasticsearch.client.internal.transport.NoNodeAvailableException: unknown node [EeCGWKRUSUaAsoaf4BZSrQ]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1256) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:303) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:111) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:481) ~[elasticsearch-8.12.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
	Suppressed: org.elasticsearch.transport.NodeNotConnectedException: [bpl-log-es3.my.localdomain.org][10.122.16.194:9300] Node not connected
		at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:283) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:874) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnectionOrFail(TransportService.java:769) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:755) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1271) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:303) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:111) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:481) ~[elasticsearch-8.12.0.jar:?]
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
		at java.lang.Thread.run(Thread.java:1583) ~[?:?]
	Suppressed: org.elasticsearch.transport.NodeNotConnectedException: [bpl-log-es3.my.localdomain.org][10.122.16.194:9300] Node not connected
		at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:283) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:874) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnectionOrFail(TransportService.java:769) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:755) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1271) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:303) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:111) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:481) ~[elasticsearch-8.12.0.jar:?]
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
		at java.lang.Thread.run(Thread.java:1583) ~[?:?]
	Suppressed: org.elasticsearch.transport.NodeNotConnectedException: [bpl-log-es3.my.localdomain.org][10.122.16.194:9300] Node not connected
		at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:283) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:874) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnectionOrFail(TransportService.java:769) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:755) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1271) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:303) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:111) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:238) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.RetryableAction.run(RetryableAction.java:89) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:322) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:225) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:172) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:306) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:278) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:240) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
		at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-02-15T14:45:11,992][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-logs-system.security-default-2024.02.08-000009][0] marking unavailable shards as stale: [pUYtfww_TiuqXgvXOjhOrw]
[2024-02-15T14:45:12,199][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.monitoring-beats-7-2024.02.15][0] marking unavailable shards as stale: [YqJ-191aSOWnoAipdIuvFg]
[2024-02-15T14:45:12,641][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.cluster_stats-default-2024.02.07-000018][0] marking unavailable shards as stale: [6FVs2-NlQ5yZwpgtNnf7lg]
[2024-02-15T14:45:12,643][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.node-default-2024.02.13-000018][0] marking unavailable shards as stale: [SgpCQ2GAQrSD7pNpoMk1gg]
[2024-02-15T14:45:12,643][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index_recovery-default-2024.02.13-000018][0] marking unavailable shards as stale: [jqovkPrISqiwDQT0UUjsGg]
[2024-02-15T14:45:12,644][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index_summary-default-2024.02.13-000018][0] marking unavailable shards as stale: [NpajjuReSZmiQcAxN4rINA]
[2024-02-15T14:45:12,644][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.node_stats-default-2024.02.13-000018][0] marking unavailable shards as stale: [W9WQakt5QqaJ9mSZqZuO5w]
[2024-02-15T14:45:12,644][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.ingest_pipeline-default-2024.02.13-000025][0] marking unavailable shards as stale: [-Q7_tkFOSCew5JRd71KDLA]
[2024-02-15T14:45:12,644][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.enrich-default-2024.02.07-000018][0] marking unavailable shards as stale: [O-9DiAFQSfOOsENBbt1DGQ]
[2024-02-15T14:45:12,645][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.shard-default-2024.02.04-000016][0] marking unavailable shards as stale: [l9Y7U_1ASa-tMsIkCBsqsw]
[2024-02-15T14:45:12,646][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index-default-2024.02.13-000020][0] marking unavailable shards as stale: [8IG3IVG9R3C9M3Jq_T4ReQ]
[2024-02-15T14:45:13,453][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.diskio-default-2024.02.08-000018][0] marking unavailable shards as stale: [2IvtYhUnSSi4_6RcmMl6ng]
[2024-02-15T14:45:13,641][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.elastic_agent-default-2024.02.08-000020][0] marking unavailable shards as stale: [ybgaFj6zQi6dIs9923Vb8Q]
[2024-02-15T14:45:13,641][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.process-default-2024.02.08-000018][0] marking unavailable shards as stale: [eBEkLiToSzuYGfOYAJ8Zfw]
[2024-02-15T14:45:13,641][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.metrics-endpoint.metadata_united_default][0] marking unavailable shards as stale: [yZZZ9JkmRHC0ZLEt_P2qBA]
[2024-02-15T14:45:13,642][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.load-default-2024.02.04-000016][0] marking unavailable shards as stale: [agErZol1SA-6XUDA1LrtZQ]
[2024-02-15T14:45:13,642][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.cpu-default-2024.02.08-000018][0] marking unavailable shards as stale: [2nqTNFgITYezlLJJf5IErg]
[2024-02-15T14:45:14,661][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-logs-elasticsearch.gc-default-2024.02.04-000007][0] marking unavailable shards as stale: [XMdWDcj2S6q1vs6biWVfiA]
[2024-02-15T14:45:15,115][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [winlogbeat-7.17.5-2024.01.22-000019][0] marking unavailable shards as stale: [R15LoXCQSfKTGU3j8Ddf_A]
[2024-02-15T14:45:15,331][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.fsstat-default-2024.02.08-000018][0] marking unavailable shards as stale: [Mr5hEl3vRW2xUVkRmYimTg]
[2024-02-15T14:45:21,054][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.background_task_utilization-default-2024.02.04-000016][0] marking unavailable shards as stale: [ooNGjwggTCeokNb6Fb8Wpg]
[2024-02-15T14:45:21,315][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.stack_monitoring.status-default-2024.02.04-000016][0] marking unavailable shards as stale: [11jEoq_oTZaar8wVKnlgKg]
[2024-02-15T14:45:21,315][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.stack_monitoring.node_actions-default-2024.02.04-000016][0] marking unavailable shards as stale: [ef16hV4_TtW7rCGSYjlI3g]
[2024-02-15T14:45:21,315][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.task_manager_metrics-default-2024.02.13-000016][0] marking unavailable shards as stale: [gFz3yA_zQ_SIZnvdAn-FFQ]
[2024-02-15T14:45:21,316][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.stack_monitoring.cluster_rules-default-2024.02.04-000016][0] marking unavailable shards as stale: [u47LUKg7ROGgFoW02RgRYA]
[2024-02-15T14:45:21,993][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-logs-elastic_agent.filebeat-default-2024.01.23-000012][0] marking unavailable shards as stale: [qgoUHsyrRmuJHNDsIxt-EQ]
[2024-02-15T14:45:22,337][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-logs-elastic_agent.metricbeat-default-2024.01.23-000012][0] marking unavailable shards as stale: [Cq__DuYnQXOMWQaMXFDHNg]
[2024-02-15T14:45:25,312][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.filesystem-default-2024.02.08-000018][0] marking unavailable shards as stale: [cjbEEE7uRm2IfbVMQRmshw]
[2024-02-15T14:45:26,202][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.metricbeat-default-2024.02.08-000020][0] marking unavailable shards as stale: [_fpFSYe0Q1y0-XRNkTqTfA]
[2024-02-15T14:45:26,415][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.filebeat-default-2024.02.08-000020][0] marking unavailable shards as stale: [JPVJzBK-SR-mZfYK3YWJ8A]
[2024-02-15T14:45:35,519][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [exchange_metrics.2023.08][0] marking unavailable shards as stale: [JKMy1p6ASoeI7TO0ZoTBFw]
[2024-02-15T14:46:12,281][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.kibana_task_manager_7.15.0_001][0] marking unavailable shards as stale: [VLYvlhgqR-GRWfwTLAZhZA]
[2024-02-15T14:46:12,542][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.tasks][0] marking unavailable shards as stale: [nGb38Q25RP6LA2ofq3Zz8A]
[2024-02-15T14:47:12,508][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2021.03-000018][2] marking unavailable shards as stale: [WayW7Ks0QrKtIZRWMb591g]
[2024-02-15T14:47:20,402][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-iis.webserver-default-2024.02.13-000015] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,402][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.load-default-2024.02.04-000016] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,403][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.uptime-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,404][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.process.summary-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,404][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.fsstat-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,405][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index_recovery-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,405][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.ml_job-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,405][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index_summary-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,406][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.filesystem-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,406][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.socket_summary-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,406][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.memory-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,407][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.elastic_agent-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,407][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.fleet_server-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,407][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.metricbeat-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,407][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-iis.application_pool-default-2024.02.13-000015] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,408][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.network-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,408][INFO][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-iis.website-default-2024.02.09-000013] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:48:53,501][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2020.10-000013][2] marking unavailable shards as stale: [rfuG_MH8TBKu69knyO6GoQ]
[2024-02-15T14:49:56,304][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2020.09-000012][1] marking unavailable shards as stale: [_2wt6o1vR5Ol4I2L-sWnrA]
[2024-02-15T14:50:47,193][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2020.08-000011][1] marking unavailable shards as stale: [ClH8gazdTGycLPPZ3ZBaVw]
[2024-02-15T14:51:04,866][WARN][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2020.07-000009][2] marking unavailable shards as stale: [sptlR4IgT4qMvN9zjWHwNA]

```

---

<div class="post-metadata">

**Author:** ![andrejze](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@andrejze](https://discuss.elastic.co/u/andrejze)\
**Post date:** [February 15, 2024, 2:51pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/8 "2024-02-15T14:51:27Z")

</div>

> [@DavidTurner](#):
>
> to wait for longer. If it's still goi

I have waited for 1 hour and the statement  
`sudo systemctl stop elasticsearch.service`  
seems to get stuck indefinitelly.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 15, 2024, 2:54pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/9 "2024-02-15T14:54:09Z")

</div>

Ok, I think we'd need to see a heap dump of the node after it's been stuck for an hour or so to understand what's going on.

---

<div class="post-metadata">

**Author:** ![andrejze](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@andrejze](https://discuss.elastic.co/u/andrejze)\
**Post date:** [February 15, 2024, 3:03pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/10 "2024-02-15T15:03:06Z")

</div>

Thank you for reply.  
Can you guide me, how can I get the heap dump, and where to post it.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 15, 2024, 3:09pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/11 "2024-02-15T15:09:58Z")

</div>

I'll send you a DM with a link you can use to send me a heap dump. To collect one, execute `jmap dump:format=b,file=heapdump.hprof $PID` where `$PID` is the process ID of the main Elasticsearch process (NB not the launcher process). You can find the `jmap` executable in the `bin` directory of your JVM.

---

<div class="post-metadata">

**Author:** ![andrejze](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@andrejze](https://discuss.elastic.co/u/andrejze)\
**Post date:** [February 15, 2024, 3:21pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/12 "2024-02-15T15:21:42Z")

</div>

Thanks, I will test again tomorrow, and send you the logs.  
If I understood correctly, the steps are:

1. (try to) stop the service on ES3
2. Wait for 1 hour
3. execute jmap cmd on ES3 and send you the heap dump?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 15, 2024, 3:22pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/13 "2024-02-15T15:22:01Z")

</div>

Correct

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 16, 2024, 10:32am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/14 "2024-02-16T10:32:30Z")

</div>

Thanks for the heap dump, the problem is clear - it's a bug in `URLRepository` with a one-line fix: [`URLRepository` should not block shutdown by DaveCTurner · Pull Request #105588 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/105588). Until you can upgrade to pick up the fix, you'll need to use `SIGKILL` to shut Elasticsearch down if it has any `URLRepository` instances registered.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 16, 2024, 10:34am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/15 "2024-02-16T10:34:34Z")

</div>

Oh actually both of your `URLRepository` instances are using a `file://` URL, you could reasonably replace them with a `FSRepository` (i.e. `type: fs`) as long as you set `readonly: true`.

---

<div class="post-metadata">

**Author:** ![andrejze](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@andrejze](https://discuss.elastic.co/u/andrejze)\
**Post date:** [February 16, 2024, 11:56am UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/16 "2024-02-16T11:56:43Z")

</div>

> [@DavidTurner](#):
>
> `FSRepository`

Are you refering to

```auto
path:
  repo:
    - /mnt/es_backups

```

in my elasticsearch.yml?  
I use this repo for backup.

How can I change the config to `FSRepository`?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 16, 2024, 12:05pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/17 "2024-02-16T12:05:28Z")

</div>

Ah sorry I'll clarify. Take a look at `GET _snapshot`: you will see two repositories with `"type": "url"` and `"url": "file://...`. These [read-only URL repositories](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-read-only-repository.html) are the only option for `http://...` URLs, but for `file://...` URLs you can do the same thing with a [shared filesystem repository](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-filesystem-repository.html), i.e. using `"type": "fs"`, as long as you set `"readonly": true`. So you can work around the bug by [unregistering](https://www.elastic.co/guide/en/elasticsearch/reference/current/delete-snapshot-repo-api.html) these repositories and then [registering](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-snapshot-repo-api.html) new repositories with `"type": "fs"` instead.

---

<div class="post-metadata">

**Author:** ![andrejze](https://avatars.discourse-cdn.com/v4/letter/a/977dab/32.png) [@andrejze](https://discuss.elastic.co/u/andrejze)\
**Post date:** [February 16, 2024, 12:33pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/18 "2024-02-16T12:33:46Z")

</div>

> [@DavidTurner](#):
>
> Ls you can do the same thin

Dear David,  
you have made my week. I have deleted all `snapshoots` of type `URL`, and ES service now stops normally.

Many thanks,  
Andrej

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [February 16, 2024, 12:45pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/19 "2024-02-16T12:45:50Z")

</div>

Great stuff, thanks for the bug report.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 15, 2024, 12:46pm UTC](https://discuss.elastic.co/t/master-not-discovered-or-elected-yet-when-stopping-elasticsearch-service/353377/20 "2024-03-15T12:46:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.