Hi!
I have a three node cluster where all nodes are maste nodes and I'm using Kibana for checking status and running queries.
Running the command:
GET _cat/nodes
Gives me:
ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
10.14.86.46 9 44 3 cdhilmrstw * STHLM-KLARA-05
10.14.86.45 20 44 20 cdhilmrstw - STHLM-KLARA-04
10.14.86.47 20 39 1 cdhilmrstw - STHLM-KLARA-06
Shutting down 04 or 06 removes them from the list above but shutting down 05 I can no longer query my cluster using Kibana, I get a time out.
The following is found in the log of 04:
[2021-09-07T16:13:22,374][TRACE][o.e.d.PeerFinder ] [STHLM-KLARA-04] startProbe(10.14.86.45:9300) not probing local node
[2021-09-07T16:13:22,375][TRACE][o.e.d.SeedHostsResolver ] [STHLM-KLARA-04] resolved host [10.14.86.45] to [10.14.86.45:9300]
[2021-09-07T16:13:22,375][TRACE][o.e.d.SeedHostsResolver ] [STHLM-KLARA-04] resolved host [10.14.86.46] to [10.14.86.46:9300]
[2021-09-07T16:13:22,375][TRACE][o.e.d.SeedHostsResolver ] [STHLM-KLARA-04] resolved host [10.14.86.47] to [10.14.86.47:9300]
[2021-09-07T16:13:22,375][TRACE][o.e.d.PeerFinder ] [STHLM-KLARA-04] probing resolved transport addresses [10.14.86.46:9300, 10.14.86.47:9300]
[2021-09-07T16:13:22,375][TRACE][o.e.d.PeerFinder ] [STHLM-KLARA-04] Peer{transportAddress=10.14.86.47:9300, discoveryNode={STHLM-KLARA-06}{EpEj69OASPeVQ3TdiZ5qEA}{Fcd1rsIgTY6-phJyChUK5g}{10.14.86.47}{10.14.86.47:9300}{cdhilmrstw}{ml.machine_memory=17178800128, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=2147483648, transform.node=true}, peersRequestInFlight=true} received PeersResponse{masterNode=Optional.empty, knownPeers=[{STHLM-KLARA-04}{sxRVZzEgRCCcRyGK5sULrQ}{UAdd75CvRpqospEfkFOcKw}{10.14.86.45}{10.14.86.45:9300}{cdhilmrstw}{ml.machine_memory=17178800128, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=2147483648, transform.node=true}], term=49}
[2021-09-07T16:13:22,375][TRACE][o.e.d.PeerFinder ] [STHLM-KLARA-04] startProbe(10.14.86.45:9300) not probing local node
[2021-09-07T16:13:23,229][TRACE][o.e.d.PeerFinder ] [STHLM-KLARA-04] startProbe(10.14.86.45:9300) not probing local node
[2021-09-07T16:13:23,370][DEBUG][o.e.d.PeerFinder ] [STHLM-KLARA-04] Peer{transportAddress=10.14.86.46:9300, discoveryNode=null, peersRequestInFlight=false} connection failed
org.elasticsearch.transport.ConnectTransportException: [][10.14.86.46:9300] connect_timeout[3s]
at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:973) ~[elasticsearch-7.11.2.jar:7.11.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) ~[elasticsearch-7.11.2.jar:7.11.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-07T16:13:23,386][TRACE][o.e.d.PeerFinder ] [STHLM-KLARA-04] Peer{transportAddress=10.14.86.47:9300, discoveryNode={STHLM-KLARA-06}{EpEj69OASPeVQ3TdiZ5qEA}{Fcd1rsIgTY6-phJyChUK5g}{10.14.86.47}{10.14.86.47:9300}{cdhilmrstw}{ml.machine_memory=17178800128, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=2147483648, transform.node=true}, peersRequestInFlight=false} requesting peers
[2021-09-07T16:13:23,386][TRACE][o.e.d.PeerFinder ] [STHLM-KLARA-04] probing master nodes from cluster state: nodes:
{STHLM-KLARA-04}{sxRVZzEgRCCcRyGK5sULrQ}{UAdd75CvRpqospEfkFOcKw}{10.14.86.45}{10.14.86.45:9300}{cdhilmrstw}{ml.machine_memory=17178800128, xpack.installed=true, transform.node=true, ml.max_open_jobs=20, ml.max_jvm_size=2147483648}, local
{STHLM-KLARA-05}{2GJuobw8RAGQE5t3J79f5Q}{N3Y6fybpRRa7B7HIqkmX4w}{10.14.86.46}{10.14.86.46:9300}{cdhilmrstw}{ml.machine_memory=17178800128, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=2147483648, transform.node=true}, master
{STHLM-KLARA-06}{EpEj69OASPeVQ3TdiZ5qEA}{Fcd1rsIgTY6-phJyChUK5g}{10.14.86.47}{10.14.86.47:9300}{cdhilmrstw}{ml.machine_memory=17178800128, ml.max_open_jobs=20, xpack.installed=true, ml.max_jvm_size=2147483648, transform.node=true}
They config files looks like this:
bootstrap.memory_lock: false
cluster.name: KLARANATET-ELASTIC-TEST
discovery.seed_hosts:
- 10.14.86.45
- 10.14.86.46
- 10.14.86.47
http.port: 9200
network.host: 10.14.86.45
node.data: true
node.ingest: true
node.master: true
node.max_local_storage_nodes: 1
node.name: STHLM-KLARA-04
path.data: D:\Elastic\ElasticSearch\Data
path.logs: D:\Elastic\ElasticSearch\Logs
transport.tcp.port: 9300
xpack.license.self_generated.type: basic
xpack.security.enabled: false
logger.org.elasticsearch.discovery: TRACE
Why is the master not changed when the 05 node is shutdown? The whole idea with a cluster solution is that one node can go down and it will still work.
What am I missing?
Thanks!
/Kristoffer