Hello,
Currently trying to upgrade a bare metal cluster, with 3 master/data nodes. All of which are running CentOS 7, and containing similar hardware. Data is stored on a separate raid volume.
I had been following the procedure listed in Rolling upgrades | Elasticsearch Guide [7.10] | Elastic, however, upon starting up the first node in the upgrade procedure, it doesn't seem to to be able to connect back to the cluster
Startup logs here; I've sanitized the node name to node-3, so that should be the full hostname, and then the last octet from the IP address.
[2024-10-10T13:24:18,809][INFO ][o.e.n.Node ] [node-3] JVM home [/usr/share/elasticsearch/jdk], using bundled JDK [true]
[2024-10-10T13:24:18,810][INFO ][o.e.n.Node ] [node-3] JVM arguments [-Xshare:auto, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -XX:+ShowCodeDetailsInExceptionMessages, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=SPI,COMPAT, -Xms1g, -Xmx1g, -XX:+UseG1GC, -XX:G1ReservePercent=25, -XX:InitiatingHeapOccupancyPercent=30, -Djava.io.tmpdir=/tmp/elasticsearch-9765179821337946878, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -Xms20g, -Xmx20g, -XX:HeapDumpPath=/mnt/data/, -XX:ErrorFile=/mnt/data/elasticlogs/hs_err_pid%p.log, -XX:MaxDirectMemorySize=10737418240, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/etc/elasticsearch, -Des.distribution.flavor=default, -Des.distribution.type=rpm, -Des.bundled_jdk=true]
[2024-10-10T13:24:22,248][INFO ][o.e.p.PluginsService ] [node-3] loaded module [aggs-matrix-stats]
[2024-10-10T13:24:22,248][INFO ][o.e.p.PluginsService ] [node-3] loaded module [analysis-common]
[2024-10-10T13:24:22,249][INFO ][o.e.p.PluginsService ] [node-3] loaded module [constant-keyword]
[2024-10-10T13:24:22,249][INFO ][o.e.p.PluginsService ] [node-3] loaded module [flattened]
[2024-10-10T13:24:22,249][INFO ][o.e.p.PluginsService ] [node-3] loaded module [frozen-indices]
[2024-10-10T13:24:22,249][INFO ][o.e.p.PluginsService ] [node-3] loaded module [ingest-common]
[2024-10-10T13:24:22,250][INFO ][o.e.p.PluginsService ] [node-3] loaded module [ingest-geoip]
[2024-10-10T13:24:22,250][INFO ][o.e.p.PluginsService ] [node-3] loaded module [ingest-user-agent]
[2024-10-10T13:24:22,250][INFO ][o.e.p.PluginsService ] [node-3] loaded module [kibana]
[2024-10-10T13:24:22,250][INFO ][o.e.p.PluginsService ] [node-3] loaded module [lang-expression]
[2024-10-10T13:24:22,251][INFO ][o.e.p.PluginsService ] [node-3] loaded module [lang-mustache]
[2024-10-10T13:24:22,251][INFO ][o.e.p.PluginsService ] [node-3] loaded module [lang-painless]
[2024-10-10T13:24:22,251][INFO ][o.e.p.PluginsService ] [node-3] loaded module [mapper-extras]
[2024-10-10T13:24:22,251][INFO ][o.e.p.PluginsService ] [node-3] loaded module [mapper-version]
[2024-10-10T13:24:22,252][INFO ][o.e.p.PluginsService ] [node-3] loaded module [parent-join]
[2024-10-10T13:24:22,252][INFO ][o.e.p.PluginsService ] [node-3] loaded module [percolator]
[2024-10-10T13:24:22,252][INFO ][o.e.p.PluginsService ] [node-3] loaded module [rank-eval]
[2024-10-10T13:24:22,252][INFO ][o.e.p.PluginsService ] [node-3] loaded module [reindex]
[2024-10-10T13:24:22,253][INFO ][o.e.p.PluginsService ] [node-3] loaded module [repositories-metering-api]
[2024-10-10T13:24:22,253][INFO ][o.e.p.PluginsService ] [node-3] loaded module [repository-url]
[2024-10-10T13:24:22,253][INFO ][o.e.p.PluginsService ] [node-3] loaded module [search-business-rules]
[2024-10-10T13:24:22,253][INFO ][o.e.p.PluginsService ] [node-3] loaded module [searchable-snapshots]
[2024-10-10T13:24:22,254][INFO ][o.e.p.PluginsService ] [node-3] loaded module [spatial]
[2024-10-10T13:24:22,254][INFO ][o.e.p.PluginsService ] [node-3] loaded module [systemd]
[2024-10-10T13:24:22,254][INFO ][o.e.p.PluginsService ] [node-3] loaded module [transform]
[2024-10-10T13:24:22,254][INFO ][o.e.p.PluginsService ] [node-3] loaded module [transport-netty4]
[2024-10-10T13:24:22,255][INFO ][o.e.p.PluginsService ] [node-3] loaded module [unsigned-long]
[2024-10-10T13:24:22,255][INFO ][o.e.p.PluginsService ] [node-3] loaded module [vectors]
[2024-10-10T13:24:22,255][INFO ][o.e.p.PluginsService ] [node-3] loaded module [wildcard]
[2024-10-10T13:24:22,255][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-analytics]
[2024-10-10T13:24:22,255][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-async]
[2024-10-10T13:24:22,256][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-async-search]
[2024-10-10T13:24:22,256][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-autoscaling]
[2024-10-10T13:24:22,256][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-ccr]
[2024-10-10T13:24:22,256][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-core]
[2024-10-10T13:24:22,257][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-data-streams]
[2024-10-10T13:24:22,257][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-deprecation]
[2024-10-10T13:24:22,257][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-enrich]
[2024-10-10T13:24:22,257][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-eql]
[2024-10-10T13:24:22,258][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-graph]
[2024-10-10T13:24:22,258][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-identity-provider]
[2024-10-10T13:24:22,258][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-ilm]
[2024-10-10T13:24:22,258][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-logstash]
[2024-10-10T13:24:22,259][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-ml]
[2024-10-10T13:24:22,259][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-monitoring]
[2024-10-10T13:24:22,259][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-ql]
[2024-10-10T13:24:22,259][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-rollup]
[2024-10-10T13:24:22,260][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-security]
[2024-10-10T13:24:22,260][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-sql]
[2024-10-10T13:24:22,260][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-stack]
[2024-10-10T13:24:22,260][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-voting-only-node]
[2024-10-10T13:24:22,260][INFO ][o.e.p.PluginsService ] [node-3] loaded module [x-pack-watcher]
[2024-10-10T13:24:22,261][INFO ][o.e.p.PluginsService ] [node-3] no plugins loaded
[2024-10-10T13:24:22,308][INFO ][o.e.e.NodeEnvironment ] [node-3] using [1] data paths, mounts [[/mnt/data (/dev/sdb1)]], net usable_space [23.8tb], net total_space [27.1tb], types [ext4]
[2024-10-10T13:24:22,309][INFO ][o.e.e.NodeEnvironment ] [node-3] heap size [20gb], compressed ordinary object pointers [true]
[2024-10-10T13:24:24,895][INFO ][o.e.n.Node ] [node-3] node name [node-3], node ID [8IcmR377Tp6tSYGBp18F6A], cluster name [elastic-1], roles [transform, master, remote_cluster_client, data, ml, data_content, data_hot, data_warm, data_cold, ingest]
[2024-10-10T13:24:29,532][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-3] [controller/29889] [Main.cc@114] controller (64 bit): Version 7.10.2 (Build 40a3af639d4698) Copyright (c) 2021 Elasticsearch BV
[2024-10-10T13:24:30,470][INFO ][o.e.x.s.a.s.FileRolesStore] [node-3] parsed [0] roles from file [/etc/elasticsearch/roles.yml]
[2024-10-10T13:24:32,712][INFO ][o.e.t.NettyAllocator ] [node-3] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2024-10-10T13:24:32,783][INFO ][o.e.d.DiscoveryModule ] [node-3] using discovery type [zen] and seed hosts providers [settings]
[2024-10-10T13:24:33,312][WARN ][o.e.g.DanglingIndicesState] [node-3] gateway.auto_import_dangling_indices is disabled, dangling indices will not be automatically detected or imported and must be managed manually
[2024-10-10T13:24:33,827][INFO ][o.e.n.Node ] [node-3] initialized
[2024-10-10T13:24:33,828][INFO ][o.e.n.Node ] [node-3] starting ...
[2024-10-10T13:24:34,299][INFO ][o.e.t.TransportService ] [node-3] publish_address {x.x.x.126:9300}, bound_addresses {x.x.x.126:9300}
[2024-10-10T13:24:36,079][INFO ][o.e.b.BootstrapChecks ] [node-3] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-10-10T13:24:46,137][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, {kCsPe_D}{kCsPe_DCR7i39X-iIQlKIg}{aQpi1af4Rt2E79-zPGLu3g}{x.x.x.125}{x.x.x.125:9300}{dim}{ml.machine_memory=33553444864, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; discovery will continue using [x.x.x.124:9300, x.x.x.125:9300] from hosts providers and [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 268670 in term 0
[2024-10-10T13:24:56,140][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, {kCsPe_D}{kCsPe_DCR7i39X-iIQlKIg}{aQpi1af4Rt2E79-zPGLu3g}{x.x.x.125}{x.x.x.125:9300}{dim}{ml.machine_memory=33553444864, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; discovery will continue using [x.x.x.124:9300, x.x.x.125:9300] from hosts providers and [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 268670 in term 0
[2024-10-10T13:25:06,142][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, {kCsPe_D}{kCsPe_DCR7i39X-iIQlKIg}{aQpi1af4Rt2E79-zPGLu3g}{x.x.x.125}{x.x.x.125:9300}{dim}{ml.machine_memory=33553444864, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; discovery will continue using [x.x.x.124:9300, x.x.x.125:9300] from hosts providers and [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 268670 in term 0
[2024-10-10T13:25:06,142][WARN ][o.e.n.Node ] [node-3] timed out while waiting for initial discovery state - timeout: 30s
[2024-10-10T13:25:06,157][INFO ][o.e.h.AbstractHttpServerTransport] [node-3] publish_address {x.x.x.126:9200}, bound_addresses {x.x.x.126:9200}
[2024-10-10T13:25:06,157][INFO ][o.e.n.Node ] [node-3] started
[2024-10-10T13:25:16,144][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, {kCsPe_D}{kCsPe_DCR7i39X-iIQlKIg}{aQpi1af4Rt2E79-zPGLu3g}{x.x.x.125}{x.x.x.125:9300}{dim}{ml.machine_memory=33553444864, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; discovery will continue using [x.x.x.124:9300, x.x.x.125:9300] from hosts providers and [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 268670 in term 0
[2024-10-10T13:25:26,146][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, {kCsPe_D}{kCsPe_DCR7i39X-iIQlKIg}{aQpi1af4Rt2E79-zPGLu3g}{x.x.x.125}{x.x.x.125:9300}{dim}{ml.machine_memory=33553444864, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; discovery will continue using [x.x.x.124:9300, x.x.x.125:9300] from hosts providers and [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 268670 in term 0
[2024-10-10T13:25:36,148][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, {kCsPe_D}{kCsPe_DCR7i39X-iIQlKIg}{aQpi1af4Rt2E79-zPGLu3g}{x.x.x.125}{x.x.x.125:9300}{dim}{ml.machine_memory=33553444864, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; discovery will continue using [x.x.x.124:9300, x.x.x.125:9300] from hosts providers and [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 268670 in term 0
[2024-10-10T13:25:36,289][INFO ][o.e.c.c.JoinHelper ] [node-3] failed to join {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} with JoinRequest{sourceNode={node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=0, optionalJoin=Optional.empty}
org.elasticsearch.transport.ReceiveTimeoutTransportException: [35fEktW][x.x.x.124:9300][internal:discovery/zen/join] request_id [11] timed out after [60043ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1083) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2024-10-10T13:25:36,297][INFO ][o.e.c.c.JoinHelper ] [node-3] failed to join {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} with JoinRequest{sourceNode={node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=0, optionalJoin=Optional.empty}
org.elasticsearch.transport.ReceiveTimeoutTransportException: [35fEktW][x.x.x.124:9300][internal:discovery/zen/join] request_id [11] timed out after [60043ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1083) [elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2024-10-10T13:25:46,150][WARN ][o.e.c.c.JoinHelper ] [node-3] last failed join attempt was 9.8s ago, failed to join {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} with JoinRequest{sourceNode={node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, minimumTerm=0, optionalJoin=Optional.empty}
org.elasticsearch.transport.ReceiveTimeoutTransportException: [35fEktW][x.x.x.124:9300][internal:discovery/zen/join] request_id [11] timed out after [60043ms]
at org.elasticsearch.transport.TransportService$TimeoutHandler.run(TransportService.java:1083) ~[elasticsearch-7.10.2.jar:7.10.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) ~[elasticsearch-7.10.2.jar:7.10.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]
[2024-10-10T13:25:46,152][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, {kCsPe_D}{kCsPe_DCR7i39X-iIQlKIg}{aQpi1af4Rt2E79-zPGLu3g}{x.x.x.125}{x.x.x.125:9300}{dim}{ml.machine_memory=33553444864, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; discovery will continue using [x.x.x.124:9300, x.x.x.125:9300] from hosts providers and [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 268670 in term 0
[2024-10-10T13:25:56,154][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-3] master not discovered yet, this node has not previously joined a bootstrapped (v7+) cluster, and [cluster.initial_master_nodes] is empty on this node: have discovered [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}, {35fEktW}{35fEktWWTieCn1VYpX9akA}{4GXu-WWOQ5iAWTZ4kTavhg}{x.x.x.124}{x.x.x.124:9300}{dim}{ml.machine_memory=33553436672, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}, {kCsPe_D}{kCsPe_DCR7i39X-iIQlKIg}{aQpi1af4Rt2E79-zPGLu3g}{x.x.x.125}{x.x.x.125:9300}{dim}{ml.machine_memory=33553444864, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}]; discovery will continue using [x.x.x.124:9300, x.x.x.125:9300] from hosts providers and [{node-3}{8IcmR377Tp6tSYGBp18F6A}{J-T5ZIOQT-u1OKDQfKD13A}{x.x.x.126}{x.x.x.126:9300}{cdhilmrstw}{ml.machine_memory=33553444864, xpack.installed=true, transform.node=true, ml.max_open_jobs=20}] from last-known cluster state; node term 0, last-accepted version 268670 in term 0
First it seems to think that its the only node in a new cluster, but after a few moments it trys to connect to the current master node only to time out.
These nodes have ports open to one another, verified via nmap, and the cluster is reporting itself as green currently.
I am not sure how to proceed. I have seen other posts talking about adding cluster.initial_master_nodes but that seems like it would be wrong, as the cluster isn't starting from new, and I'm afraid that will erase data.
I can provide some more logging or other info if needed