Master not discovered or elected yet, when stopping Elasticsearch service

andrejze · February 15, 2024, 2:13pm

OS: Debian 10 (buster), 4.19.0-25-amd64 #1 SMP Debian 4.19.289-2 (2023-08-08)
Elasticsearch: 8.12.0
Cluster: 3 nodes, all master eligible

When stopping Elasticsearch service on any node, operation gets stuck with error :

[2024-02-15T14:45:21,656][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html

It continues to show this error, until the java process is killed with SIGKILL signal.

My elasticsearch.yml:

cluster.name: bpl-log-cluster
node.name: bpl-log-es3.my.localdomain.org
# cluster.initial_master_nodes: ["bpl-log-es1.my.localdomain.org", "bpl-log-es2.my.localdomain.org", "bpl-log-es3.my.localdomain.org"]
discovery.seed_hosts: ["bpl-log-es1.my.localdomain.org", "bpl-log-es2.my.localdomain.org", "bpl-log-es3.my.localdomain.org"]

network.host: 0.0.0.0

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /etc/elasticsearch/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /etc/elasticsearch/elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/bpl-log-es_chain-nopass.pfx
xpack.security.http.ssl.truststore.path: "/etc/elasticsearch/cert_ca.jks"

path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch

andrejze · February 15, 2024, 2:14pm

Elasticsearch cluster log:

[2024-02-15T14:45:11,232][INFO ][o.e.n.Node               ] [bpl-log-es3.my.localdomain.org] stopping ...
[2024-02-15T14:45:11,233][INFO ][o.e.x.w.WatcherService   ] [bpl-log-es3.my.localdomain.org] stopping watch service, reason [shutdown initiated]
[2024-02-15T14:45:11,233][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [bpl-log-es3.my.localdomain.org] [controller/28297] [Main.cc@176] ML controller exiting
[2024-02-15T14:45:11,234][INFO ][o.e.x.w.WatcherLifeCycleService] [bpl-log-es3.my.localdomain.org] watcher has stopped and shutdown
[2024-02-15T14:45:11,235][INFO ][o.e.x.m.p.NativeController] [bpl-log-es3.my.localdomain.org] Native controller process has stopped - no new native processes can be started
[2024-02-15T14:45:11,651][INFO ][o.e.c.c.Coordinator      ] [bpl-log-es3.my.localdomain.org] master node [{bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] disconnected, restarting discovery
[2024-02-15T14:45:13,681][INFO ][o.e.n.Node               ] [bpl-log-es3.my.localdomain.org] stopped
[2024-02-15T14:45:13,681][INFO ][o.e.n.Node               ] [bpl-log-es3.my.localdomain.org] closing ...
[2024-02-15T14:45:21,656][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:45:31,658][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:45:41,660][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:45:51,662][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:01,663][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:11,665][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:21,667][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:31,669][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:41,671][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:46:51,673][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:01,674][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:11,676][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:21,678][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:31,679][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:41,681][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:47:51,683][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:48:01,685][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:48:11,686][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bpl-log-es3.my.localdomain.org] master not discovered or elected yet, an election requires at least 2 nodes with ids from [ykzajm2GRSe3oJKNHgk8Hg, EeCGWKRUSUaAsoaf4BZSrQ, sjk7jww6R7SN8I2t3oRZkQ], have only discovered non-quorum [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}]; discovery will continue using [10.122.16.192:9300, 10.122.16.193:9300, 127.0.1.1:9300] from hosts providers and [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}, {bpl-log-es2.my.localdomain.org}{ykzajm2GRSe3oJKNHgk8Hg}{A9LNzrpVR-CV5Vaj-vX_gg}{bpl-log-es2.my.localdomain.org}{10.122.16.193}{10.122.16.193:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 244, last-accepted version 262057 in term 244; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-15T14:48:12,793][WARN ][o.e.d.PeerFinder         ] [bpl-log-es3.my.localdomain.org] address [10.122.16.192:9300], node [null], requesting [false] discovery result: [][10.122.16.192:9300] connection manager is closed
[2024-02-15T14:48:12,794][WARN ][o.e.d.PeerFinder         ] [bpl-log-es3.my.localdomain.org] address [10.122.16.193:9300], node [null], requesting [false] discovery result: [][10.122.16.193:9300] connection manager is closed
[2024-02-15T14:48:13,795][WARN ][o.e.d.PeerFinder         ] [bpl-log-es3.my.localdomain.org] address [10.122.16.192:9300], node [null], requesting [false] discovery result: [][10.122.16.192:9300] connection manager is closed
[2024-02-15T14:48:13,796][WARN ][o.e.d.PeerFinder         ] [bpl-log-es3.my.localdomain.org] address [10.122.16.193:9300], node [null], requesting [false] discovery result: [][10.122.16.193:9300] connection manager is closed

leandrojmp · February 15, 2024, 2:23pm

What do you have in the other nodes logs?

If I understood correctly you stopped the es1 node and these logs are from the es3 nodes, it is saying that it cannot form a quorum.

What do you have in the es2 node logs?

DavidTurner · February 15, 2024, 2:37pm

These logs indicate you have stopped bpl-log-es1 and bpl-log-es2 so it's expected that the cluster cannot form.

DavidTurner · February 15, 2024, 2:39pm

Oh wait sorry that's not right, you're trying to shut down bpl-log-es3. I think you probably just need to wait for longer. If it's still going after an hour or so, we might need to look at a heap dump to understand what is stuck.

DavidTurner · February 15, 2024, 2:44pm

I opened `ClusterFormationFailureHandler` shouldn't log anything once the node is stopping · Issue #105559 · elastic/elasticsearch · GitHub because this is a logging bug. For now, you can ignore these warning logs.

andrejze · February 15, 2024, 2:48pm

In the above scenario I have stopped the service on ES3 node and logs are from ES3 node. Master is ES1:

name                      version node.role   diskUsedPercent master nodeRole    nodeId                 http
bpl-log-es2.my.localdomain.org 8.12.0  cdfhilmrstw           69.77 -      cdfhilmrstw ykzajm2GRSe3oJKNHgk8Hg 10.122.16.193:9200
bpl-log-es1.my.localdomain.org 8.12.0  cdfhilmrstw           57.87 *      cdfhilmrstw sjk7jww6R7SN8I2t3oRZkQ 10.122.16.192:9200
bpl-log-es3.my.localdomain.org 8.12.0  cdfhilmrstw           32.17 -      cdfhilmrstw EeCGWKRUSUaAsoaf4BZSrQ 10.122.16.194:9200

Both ES1 and ES2 were running and where accessible on port 9200 and 9300 from ES3 tested with nc and telnet.

Here is log from ES2 from the same time:

[2024-02-15T14:45:11,652][INFO ][o.e.t.ClusterConnectionManager] [bpl-log-es2.my.localdomain.org] transport connection to [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] closed by remote

[2024-02-15T14:45:11,835][INFO ][o.e.c.s.ClusterApplierService] [bpl-log-es2.my.localdomain.org] removed {{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}}, term: 244, version: 262058, reason: ApplyCommitRequest{term=244, version=262058, sourceNode={bpl-log-es1.my.localdomain.org}{sjk7jww6R7SN8I2t3oRZkQ}{GuPWU5dWSQuGYpWF_-ZpoQ}{bpl-log-es1.my.localdomain.org}{10.122.16.192}{10.122.16.192:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}{ml.machine_memory=33731747840, xpack.installed=true, transform.config_version=10.0.0, ml.config_version=12.0.0, ml.max_jvm_size=17179869184, ml.allocated_processors_double=8.0, ml.allocated_processors=8}}

[2024-02-15T14:45:11,867][INFO ][o.e.i.s.IndexShard ] [bpl-log-es2.my.localdomain.org] [.tasks][0] primary-replica resync completed with 0 operations

[2024-02-15T14:45:11,867][INFO ][o.e.i.s.IndexShard ] [bpl-log-es2.my.localdomain.org] [.kibana_task_manager_7.15.0_001][0] primary-replica resync completed with 0 operations

[2024-02-15T14:47:20,337][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.memory-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,339][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.network-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,339][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.node-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,340][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.uptime-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,341][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.socket_summary-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,341][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.process.summary-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,343][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-iis.application_pool-default-2024.02.13-000015] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,343][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.cpu-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,344][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.node_stats-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,346][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.diskio-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,347][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elastic_agent.fleet_server-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,348][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.ml_job-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,348][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index-default-2024.02.13-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,349][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elasticsearch.ingest_pipeline-default-2024.02.13-000025] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,350][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-elastic_agent.filebeat-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,351][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-system.process-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,351][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-iis.website-default-2024.02.09-000013] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

[2024-02-15T14:47:20,352][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es2.my.localdomain.org] [.ds-metrics-iis.webserver-default-2024.02.13-000015] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]

Logs from ES1:

[2024-02-15T14:42:31,060][INFO ][o.e.c.m.MetadataMappingService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.task_manager_metrics-default-2024.02.13-000016/-K-xDhV8QraPqQvpoZpjjA] update_mapping [_doc]
[2024-02-15T14:45:11,654][INFO ][o.e.t.ClusterConnectionManager] [bpl-log-es1.my.localdomain.org] transport connection to [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] closed by remote
[2024-02-15T14:45:11,660][INFO ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] current.health="YELLOW" message="Cluster health status changed from [GREEN] to [YELLOW] (reason: [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008} reason: disconnected])." previous.health="GREEN" reason="{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008} reason: disconnected"
[2024-02-15T14:45:11,662][INFO ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] updating number_of_replicas to [1] for indices [.apm-source-map]
[2024-02-15T14:45:11,762][INFO ][o.e.c.s.MasterService    ] [bpl-log-es1.my.localdomain.org] node-left[{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008} reason: disconnected], term: 244, version: 262058, delta: removed {{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}}
[2024-02-15T14:45:11,903][INFO ][o.e.c.s.ClusterApplierService] [bpl-log-es1.my.localdomain.org] removed {{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}}, term: 244, version: 262058, reason: Publication{term=244, version=262058}
[2024-02-15T14:45:11,971][INFO ][o.e.c.r.DelayedAllocationService] [bpl-log-es1.my.localdomain.org] scheduling reroute for delayed shards in [59.6s] (277 delayed shards)
[2024-02-15T14:45:11,971][INFO ][o.e.c.c.NodeLeftExecutor ] [bpl-log-es1.my.localdomain.org] node-left: [{bpl-log-es3.my.localdomain.org}{EeCGWKRUSUaAsoaf4BZSrQ}{B1qcJ-z2SUaMN-kygi80bA}{bpl-log-es3.my.localdomain.org}{10.122.16.194}{10.122.16.194:9300}{cdfhilmrstw}{8.12.0}{7000099-8500008}] with reason [disconnected]
[2024-02-15T14:45:11,976][WARN ][o.e.a.b.TransportShardBulkAction] [bpl-log-es1.my.localdomain.org] [[.ds-logs-system.security-default-2024.02.08-000009][0]] failed to perform indices:data/write/bulk[s] on replica [.ds-logs-system.security-default-2024.02.08-000009][0], node[EeCGWKRUSUaAsoaf4BZSrQ], [R], s[STARTED], a[id=pUYtfww_TiuqXgvXOjhOrw], failed_attempts[0]
org.elasticsearch.client.internal.transport.NoNodeAvailableException: unknown node [EeCGWKRUSUaAsoaf4BZSrQ]
	at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1256) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:303) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:111) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:481) ~[elasticsearch-8.12.0.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
	Suppressed: org.elasticsearch.transport.NodeNotConnectedException: [bpl-log-es3.my.localdomain.org][10.122.16.194:9300] Node not connected
		at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:283) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:874) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnectionOrFail(TransportService.java:769) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:755) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1271) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:303) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:111) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:481) ~[elasticsearch-8.12.0.jar:?]
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
		at java.lang.Thread.run(Thread.java:1583) ~[?:?]
	Suppressed: org.elasticsearch.transport.NodeNotConnectedException: [bpl-log-es3.my.localdomain.org][10.122.16.194:9300] Node not connected
		at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:283) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:874) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnectionOrFail(TransportService.java:769) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:755) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1271) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:303) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:111) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:481) ~[elasticsearch-8.12.0.jar:?]
		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) ~[?:?]
		at java.util.concurrent.FutureTask.run(FutureTask.java:317) ~[?:?]
		at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
		at java.lang.Thread.run(Thread.java:1583) ~[?:?]
	Suppressed: org.elasticsearch.transport.NodeNotConnectedException: [bpl-log-es3.my.localdomain.org][10.122.16.194:9300] Node not connected
		at org.elasticsearch.transport.ClusterConnectionManager.getConnection(ClusterConnectionManager.java:283) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnection(TransportService.java:874) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.getConnectionOrFail(TransportService.java:769) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:755) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicasProxy.performOn(TransportReplicationAction.java:1271) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation$3.tryAction(ReplicationOperation.java:303) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.RetryableAction$1.doRun(RetryableAction.java:111) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:238) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.RetryableAction.run(RetryableAction.java:89) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplica(ReplicationOperation.java:322) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation.performOnReplicas(ReplicationOperation.java:225) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.ReplicationOperation.handlePrimaryResult(ReplicationOperation.java:172) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:306) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction$2.finishRequest(TransportShardBulkAction.java:278) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:240) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:291) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:142) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:74) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0.jar:?]
		at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
		at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-02-15T14:45:11,992][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-logs-system.security-default-2024.02.08-000009][0] marking unavailable shards as stale: [pUYtfww_TiuqXgvXOjhOrw]
[2024-02-15T14:45:12,199][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.monitoring-beats-7-2024.02.15][0] marking unavailable shards as stale: [YqJ-191aSOWnoAipdIuvFg]
[2024-02-15T14:45:12,641][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.cluster_stats-default-2024.02.07-000018][0] marking unavailable shards as stale: [6FVs2-NlQ5yZwpgtNnf7lg]
[2024-02-15T14:45:12,643][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.node-default-2024.02.13-000018][0] marking unavailable shards as stale: [SgpCQ2GAQrSD7pNpoMk1gg]
[2024-02-15T14:45:12,643][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index_recovery-default-2024.02.13-000018][0] marking unavailable shards as stale: [jqovkPrISqiwDQT0UUjsGg]
[2024-02-15T14:45:12,644][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index_summary-default-2024.02.13-000018][0] marking unavailable shards as stale: [NpajjuReSZmiQcAxN4rINA]
[2024-02-15T14:45:12,644][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.node_stats-default-2024.02.13-000018][0] marking unavailable shards as stale: [W9WQakt5QqaJ9mSZqZuO5w]
[2024-02-15T14:45:12,644][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.ingest_pipeline-default-2024.02.13-000025][0] marking unavailable shards as stale: [-Q7_tkFOSCew5JRd71KDLA]
[2024-02-15T14:45:12,644][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.enrich-default-2024.02.07-000018][0] marking unavailable shards as stale: [O-9DiAFQSfOOsENBbt1DGQ]
[2024-02-15T14:45:12,645][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.shard-default-2024.02.04-000016][0] marking unavailable shards as stale: [l9Y7U_1ASa-tMsIkCBsqsw]
[2024-02-15T14:45:12,646][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index-default-2024.02.13-000020][0] marking unavailable shards as stale: [8IG3IVG9R3C9M3Jq_T4ReQ]
[2024-02-15T14:45:13,453][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.diskio-default-2024.02.08-000018][0] marking unavailable shards as stale: [2IvtYhUnSSi4_6RcmMl6ng]
[2024-02-15T14:45:13,641][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.elastic_agent-default-2024.02.08-000020][0] marking unavailable shards as stale: [ybgaFj6zQi6dIs9923Vb8Q]
[2024-02-15T14:45:13,641][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.process-default-2024.02.08-000018][0] marking unavailable shards as stale: [eBEkLiToSzuYGfOYAJ8Zfw]
[2024-02-15T14:45:13,641][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.metrics-endpoint.metadata_united_default][0] marking unavailable shards as stale: [yZZZ9JkmRHC0ZLEt_P2qBA]
[2024-02-15T14:45:13,642][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.load-default-2024.02.04-000016][0] marking unavailable shards as stale: [agErZol1SA-6XUDA1LrtZQ]
[2024-02-15T14:45:13,642][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.cpu-default-2024.02.08-000018][0] marking unavailable shards as stale: [2nqTNFgITYezlLJJf5IErg]
[2024-02-15T14:45:14,661][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-logs-elasticsearch.gc-default-2024.02.04-000007][0] marking unavailable shards as stale: [XMdWDcj2S6q1vs6biWVfiA]
[2024-02-15T14:45:15,115][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [winlogbeat-7.17.5-2024.01.22-000019][0] marking unavailable shards as stale: [R15LoXCQSfKTGU3j8Ddf_A]
[2024-02-15T14:45:15,331][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.fsstat-default-2024.02.08-000018][0] marking unavailable shards as stale: [Mr5hEl3vRW2xUVkRmYimTg]
[2024-02-15T14:45:21,054][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.background_task_utilization-default-2024.02.04-000016][0] marking unavailable shards as stale: [ooNGjwggTCeokNb6Fb8Wpg]
[2024-02-15T14:45:21,315][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.stack_monitoring.status-default-2024.02.04-000016][0] marking unavailable shards as stale: [11jEoq_oTZaar8wVKnlgKg]
[2024-02-15T14:45:21,315][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.stack_monitoring.node_actions-default-2024.02.04-000016][0] marking unavailable shards as stale: [ef16hV4_TtW7rCGSYjlI3g]
[2024-02-15T14:45:21,315][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.task_manager_metrics-default-2024.02.13-000016][0] marking unavailable shards as stale: [gFz3yA_zQ_SIZnvdAn-FFQ]
[2024-02-15T14:45:21,316][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-kibana.stack_monitoring.cluster_rules-default-2024.02.04-000016][0] marking unavailable shards as stale: [u47LUKg7ROGgFoW02RgRYA]
[2024-02-15T14:45:21,993][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-logs-elastic_agent.filebeat-default-2024.01.23-000012][0] marking unavailable shards as stale: [qgoUHsyrRmuJHNDsIxt-EQ]
[2024-02-15T14:45:22,337][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-logs-elastic_agent.metricbeat-default-2024.01.23-000012][0] marking unavailable shards as stale: [Cq__DuYnQXOMWQaMXFDHNg]
[2024-02-15T14:45:25,312][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.filesystem-default-2024.02.08-000018][0] marking unavailable shards as stale: [cjbEEE7uRm2IfbVMQRmshw]
[2024-02-15T14:45:26,202][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.metricbeat-default-2024.02.08-000020][0] marking unavailable shards as stale: [_fpFSYe0Q1y0-XRNkTqTfA]
[2024-02-15T14:45:26,415][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.filebeat-default-2024.02.08-000020][0] marking unavailable shards as stale: [JPVJzBK-SR-mZfYK3YWJ8A]
[2024-02-15T14:45:35,519][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [exchange_metrics.2023.08][0] marking unavailable shards as stale: [JKMy1p6ASoeI7TO0ZoTBFw]
[2024-02-15T14:46:12,281][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.kibana_task_manager_7.15.0_001][0] marking unavailable shards as stale: [VLYvlhgqR-GRWfwTLAZhZA]
[2024-02-15T14:46:12,542][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [.tasks][0] marking unavailable shards as stale: [nGb38Q25RP6LA2ofq3Zz8A]
[2024-02-15T14:47:12,508][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2021.03-000018][2] marking unavailable shards as stale: [WayW7Ks0QrKtIZRWMb591g]
[2024-02-15T14:47:20,402][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-iis.webserver-default-2024.02.13-000015] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,402][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.load-default-2024.02.04-000016] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,403][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.uptime-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,404][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.process.summary-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,404][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.fsstat-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,405][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index_recovery-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,405][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.ml_job-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,405][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elasticsearch.stack_monitoring.index_summary-default-2024.02.13-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,406][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.filesystem-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,406][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.socket_summary-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,406][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.memory-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,407][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.elastic_agent-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,407][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.fleet_server-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,407][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-elastic_agent.metricbeat-default-2024.02.08-000020] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,407][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-iis.application_pool-default-2024.02.13-000015] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,408][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-system.network-default-2024.02.08-000018] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:47:20,408][INFO ][o.e.c.s.IndexScopedSettings] [bpl-log-es1.my.localdomain.org] [.ds-metrics-iis.website-default-2024.02.09-000013] updating [index.time_series.end_time] from [2024-02-15T15:47:20.000Z] to [2024-02-15T15:52:20.000Z]
[2024-02-15T14:48:53,501][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2020.10-000013][2] marking unavailable shards as stale: [rfuG_MH8TBKu69knyO6GoQ]
[2024-02-15T14:49:56,304][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2020.09-000012][1] marking unavailable shards as stale: [_2wt6o1vR5Ol4I2L-sWnrA]
[2024-02-15T14:50:47,193][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2020.08-000011][1] marking unavailable shards as stale: [ClH8gazdTGycLPPZ3ZBaVw]
[2024-02-15T14:51:04,866][WARN ][o.e.c.r.a.AllocationService] [bpl-log-es1.my.localdomain.org] [remoteaccess-2020.07-000009][2] marking unavailable shards as stale: [sptlR4IgT4qMvN9zjWHwNA]

andrejze · February 15, 2024, 2:51pm

I have waited for 1 hour and the statement
sudo systemctl stop elasticsearch.service
seems to get stuck indefinitelly.

DavidTurner · February 15, 2024, 2:54pm

Ok, I think we'd need to see a heap dump of the node after it's been stuck for an hour or so to understand what's going on.

andrejze · February 15, 2024, 3:03pm

Thank you for reply.
Can you guide me, how can I get the heap dump, and where to post it.

DavidTurner · February 15, 2024, 3:09pm

I'll send you a DM with a link you can use to send me a heap dump. To collect one, execute jmap dump:format=b,file=heapdump.hprof $PID where $PID is the process ID of the main Elasticsearch process (NB not the launcher process). You can find the jmap executable in the bin directory of your JVM.

andrejze · February 15, 2024, 3:21pm

Thanks, I will test again tomorrow, and send you the logs.
If I understood correctly, the steps are:

(try to) stop the service on ES3
Wait for 1 hour
execute jmap cmd on ES3 and send you the heap dump?

DavidTurner · February 15, 2024, 3:22pm

Correct

DavidTurner · February 16, 2024, 10:32am

Thanks for the heap dump, the problem is clear - it's a bug in URLRepository with a one-line fix: `URLRepository` should not block shutdown by DaveCTurner · Pull Request #105588 · elastic/elasticsearch · GitHub. Until you can upgrade to pick up the fix, you'll need to use SIGKILL to shut Elasticsearch down if it has any URLRepository instances registered.

DavidTurner · February 16, 2024, 10:34am

Oh actually both of your URLRepository instances are using a file:// URL, you could reasonably replace them with a FSRepository (i.e. type: fs) as long as you set readonly: true.

andrejze · February 16, 2024, 11:56am

Are you refering to

path:
  repo:
    - /mnt/es_backups

in my elasticsearch.yml?
I use this repo for backup.

How can I change the config to FSRepository?

DavidTurner · February 16, 2024, 12:05pm

Ah sorry I'll clarify. Take a look at GET _snapshot: you will see two repositories with "type": "url" and "url": "file://.... These read-only URL repositories are the only option for http://... URLs, but for file://... URLs you can do the same thing with a shared filesystem repository, i.e. using "type": "fs", as long as you set "readonly": true. So you can work around the bug by unregistering these repositories and then registering new repositories with "type": "fs" instead.

andrejze · February 16, 2024, 12:33pm

Dear David,
you have made my week. I have deleted all snapshoots of type URL, and ES service now stops normally.

Many thanks,
Andrej

DavidTurner · February 16, 2024, 12:45pm

Great stuff, thanks for the bug report.

system · March 15, 2024, 12:46pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.