Windows two Node Cluster stuck on tring to determine master

Elastic Stack Elasticsearch
1

I have been trying to install a new two node cluster on two windows servers and no matter what I have tried I get a situation where the two nodes do not seem to be able to determine which should be the master. I have tried starting from scratch several times but get the same result. I am not sure what I am doing wrong. I unzipped the elasticsearch into a folder and edited the elasticsearch.yml file on both machines. I created a ca and certificate which I copied to both machines and stored the passwords in the keystore.
The elasticsearch.yml for the first node is below

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: bytelink-network
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: bytelink-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 10.30.1.12
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["10.30.1.12", "10.30.1.13"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["bytelink-1", "bytelink-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#
# ---------------------------------- Security -----------------------------------
#
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: D:\Elastic ELK Stack\elasticsearch-8.11.3\config\certs\elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: D:\Elastic ELK Stack\elasticsearch-8.11.3\config\certs\elastic-certificates.p12

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: D:\Elastic ELK Stack\elasticsearch-8.11.3\config\certs\http.p12

The elasticsearch.yml for the second node is below

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: bytelink-network
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: bytelink-2
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 10.30.1.13
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["10.30.1.12", "10.30.1.13"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["bytelink-1", "bytelink-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#
# ---------------------------------- Security -----------------------------------
#
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: 'D:\Elastic ELK Stack\elasticsearch-8.11.3\config\certs\elastic-certificates.p12'
xpack.security.transport.ssl.truststore.path: 'D:\Elastic ELK Stack\elasticsearch-8.11.3\config\certs\elastic-certificates.p12'

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: 'D:\Elastic ELK Stack\elasticsearch-8.11.3\config\certs\http.p12'

The internal IP of both machines is 10.30.1.12 and 10.30.1.13 respectively.
The message I get for Node 1 is as follows

[2024-01-09T22:15:11,678][WARN ][o.e.c.c.JoinHelper       ] [bytelink-1] last failed join attempt was 9.7s ago, failed to join {bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{ml.allocated_processors=32, ml.machine_memory=137338658816, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=33285996544, ml.allocated_processors_double=32.0} with JoinRequest{sourceNode={bytelink-1}{hPROK0g2S0yyaSW6CsyaXg}{7ac3SqZDQ4y4n0VDGb-F0w}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{ml.allocated_processors=4, ml.machine_memory=103012937728, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=33285996544, ml.allocated_processors_double=4.0}, compatibilityVersions=CompatibilityVersions[transportVersion=8512001, systemIndexMappingsVersion={.triggered_watches=MappingsVersion[version=1, hash=-502826165], .secrets-inference=MappingsVersion[version=1, hash=-1434574148], .fleet-agents-7=MappingsVersion[version=1, hash=-644511608], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .ml-config=MappingsVersion[version=1, hash=1455005892], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=1, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-1=MappingsVersion[version=1, hash=1647955624], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-840564854], .fleet-actions-7=MappingsVersion[version=1, hash=961753072], .tasks=MappingsVersion[version=0, hash=-945584329], .ml-meta=MappingsVersion[version=1, hash=976416370], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .security-7=MappingsVersion[version=1, hash=-1061511639], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=2, hash=-1534099917], .inference=MappingsVersion[version=1, hash=645017346], .async-search=MappingsVersion[version=0, hash=-1403744380], .fleet-policies-7=MappingsVersion[version=1, hash=762674907], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], minimumTerm=145, optionalJoin=Optional[Join{term=146, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={bytelink-1}{hPROK0g2S0yyaSW6CsyaXg}{7ac3SqZDQ4y4n0VDGb-F0w}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{ml.allocated_processors=4, ml.machine_memory=103012937728, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=33285996544, ml.allocated_processors_double=4.0}, targetNode={bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{ml.allocated_processors=32, ml.machine_memory=137338658816, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=33285996544, ml.allocated_processors_double=32.0}}]}org.elasticsearch.transport.ConnectTransportException: [bytelink-2][10.30.1.13:9300] failed to acquire connection
        at org.elasticsearch.server@8.11.3/org.elasticsearch.cluster.coordination.JoinHelper$1.onFailure(JoinHelper.java:345)
        at org.elasticsearch.server@8.11.3/org.elasticsearch.action.ActionListenerImplementations.safeAcceptException(ActionListenerImplementations.java:62)
        at org.elasticsearch.server@8.11.3/org.elasticsearch.action.ActionListenerImplementations.safeOnFailure(ActionListenerImplementations.java:73)
        at org.elasticsearch.server@8.11.3/org.elasticsearch.action.DelegatingActionListener.onFailure(DelegatingActionListener.java:27)
        at org.elasticsearch.server@8.11.3/org.elasticsearch.action.support.ContextPreservingActionListener.onFailure(ContextPreservingActionListener.java:39)

The message I get for Node 2 is as follows

[2024-01-09T22:17:50,532][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bytelink-2] master not discovered or elected yet, an election requires two nodes with ids [6lgj7xkVQWOs22q29YKnHA, hPROK0g2S0yyaSW6CsyaXg], have discovered possible quorum [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}, {bytelink-1}{hPROK0g2S0yyaSW6CsyaXg}{7ac3SqZDQ4y4n0VDGb-F0w}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}]; discovery will continue using [10.30.1.12:9300] from hosts providers and [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}] from last-known cluster state; node term 175, last-accepted version 0 in term 0; joining [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}] in term [175] has status [waiting for response] after [2.8s/2810ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html
[2024-01-09T22:18:00,536][WARN ][o.e.c.c.JoinHelper       ] [bytelink-2] last failed join attempt was 2.3s ago, failed to join {bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{ml.allocated_processors_double=32.0, ml.max_jvm_size=33285996544, ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=137338658816, ml.allocated_processors=32} with JoinRequest{sourceNode={bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{ml.allocated_processors_double=32.0, ml.max_jvm_size=33285996544, ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=137338658816, ml.allocated_processors=32}, compatibilityVersions=CompatibilityVersions[transportVersion=8512001, systemIndexMappingsVersion={.secrets-inference=MappingsVersion[version=1, hash=-1434574148], .triggered_watches=MappingsVersion[version=1, hash=-502826165], .fleet-agents-7=MappingsVersion[version=1, hash=-644511608], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .ml-config=MappingsVersion[version=1, hash=1455005892], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=1, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-1=MappingsVersion[version=1, hash=1647955624], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-840564854], .fleet-actions-7=MappingsVersion[version=1, hash=961753072], .tasks=MappingsVersion[version=0, hash=-945584329], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .ml-meta=MappingsVersion[version=1, hash=976416370], .security-7=MappingsVersion[version=1, hash=-1061511639], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=2, hash=-1534099917], .async-search=MappingsVersion[version=0, hash=-1403744380], .inference=MappingsVersion[version=1, hash=645017346], .fleet-policies-7=MappingsVersion[version=1, hash=762674907], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], minimumTerm=175, optionalJoin=Optional[Join{term=176, lastAcceptedTerm=0, lastAcceptedVersion=0, sourceNode={bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{ml.allocated_processors_double=32.0, ml.max_jvm_size=33285996544, ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=137338658816, ml.allocated_processors=32}, targetNode={bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{ml.allocated_processors_double=32.0, ml.max_jvm_size=33285996544, ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=137338658816, ml.allocated_processors=32}}]}org.elasticsearch.transport.RemoteTransportException: [bytelink-2][10.30.1.13:9300][internal:cluster/coordination/join]

See logs for more details.

[2024-01-09T22:18:00,539][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bytelink-2] master not discovered or elected yet, an election requires two nodes with ids [6lgj7xkVQWOs22q29YKnHA, hPROK0g2S0yyaSW6CsyaXg], have discovered possible quorum [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}, {bytelink-1}{hPROK0g2S0yyaSW6CsyaXg}{7ac3SqZDQ4y4n0VDGb-F0w}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}]; discovery will continue using [10.30.1.12:9300] from hosts providers and [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}] from last-known cluster state; node term 177, last-accepted version 0 in term 0; joining [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{R7Eeow3xR4i1p4y-6jRLNQ}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}] in term [177] has status [waiting for response] after [2.2s/2211ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html

Thanks in advance for taking the time to look into it

2

Since a two node cluster can not be resilient to failures, it is recommended that you set just one of the nodes as master eligible and leave the other node as just a data node.

I would recommend that you set just one of them as the master.

For one of them you should use:

node.roles: [ master, data, ingest ]

and for the other you should use:

node.roles: [ data, ingest ].

Change the settings and restart both nodes.

3

Thank you for your reply. I added the lines in the appropriate files with bytelink-1 being master and bytelink-2 as the data. I then restarted and got a warning on master to delete the cluster.initial.master_nodes setting which I did on both and then restarted both. I now get the following message repeated on master:

[2024-01-10T10:12:06,593][WARN ][o.e.c.c.Coordinator      ] [bytelink-1] This node is a fully-formed single-node cluster with cluster UUID [U_YR3FXbR1Gm09to-vmrxA], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[10.30.1.12, 10.30.1.13]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message.
[2024-01-10T10:12:15,919][WARN ][o.e.c.c.Coordinator      ] [bytelink-1] received join request from [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{f0aE9FnmQSmgZhXLaDGToA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, ml.config_version=11.0.0, xpack.installed=true}] but could not connect back to the joining nodeorg.elasticsearch.transport.ConnectTransportException: [bytelink-2][10.30.1.13:9300] connect_exception
        at org.elasticsearch.server@8.11.3/org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1156)
        at org.elasticsearch.server@8.11.3/org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:322)
        at org.elasticsearch.server@8.11.3/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:234)
        at org.elasticsearch.server@8.11.3/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259)
        at org.elasticsearch.server@8.11.3/org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178)

and the following repeated on the data:

[2024-01-10T10:21:00,991][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bytelink-2] master not discovered yet: have discovered [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{f0aE9FnmQSmgZhXLaDGToA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}, {bytelink-1}{hPROK0g2S0yyaSW6CsyaXg}{r3XML0UWT3Wj2sR4JAbrfw}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{dim}{8.11.3}{7000099-8500003}]; discovery will continue using [10.30.1.12:9300] from hosts providers and [] from last-known cluster state; node term 7389, last-accepted version 0 in term 0; joining [{bytelink-1}{hPROK0g2S0yyaSW6CsyaXg}{r3XML0UWT3Wj2sR4JAbrfw}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{dim}{8.11.3}{7000099-8500003}] in term [7389] has status [waiting for response] after [9m/545962ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html
[2024-01-10T10:21:00,996][WARN ][o.e.c.c.CoordinationDiagnosticsService] [bytelink-2] Exception in remote request to master{bytelink-1}{hPROK0g2S0yyaSW6CsyaXg}{r3XML0UWT3Wj2sR4JAbrfw}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{dim}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}org.elasticsearch.transport.ReceiveTimeoutTransportException: [bytelink-1][10.30.1.12:9300][internal:cluster/coordination_diagnostics/info] request_id [266] timed out after [9833ms]

See logs for more details.

[2024-01-10T10:21:01,476][WARN ][o.e.d.PeerFinder         ] [bytelink-2] address [10.30.1.12:9300], node [{bytelink-1}{hPROK0g2S0yyaSW6CsyaXg}{r3XML0UWT3Wj2sR4JAbrfw}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{dim}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}], requesting [false] peers request failedorg.elasticsearch.transport.ReceiveTimeoutTransportException: [bytelink-1][10.30.1.12:9300][internal:discovery/request_peers] request_id [269] timed out after [3009ms]

Should I change the line below as well to just show bytelink-1 ip address only

discovery.seed_hosts: ["10.30.1.12", "10.30.1.13"]
#

Thank you again

4

Yes, as explained in the documentation discovery.seed_hosts should have only the addresses of master-eligible nodes, since only the first node is master-eligible, only this node needs to be present.

The same thing for cluster.initial_master_nodes, but according to your logs the bytelink-1 node already formed as a single-node cluster, so you can remove this line as well and the bytelink-2 should be able to join the cluster as a data node only.

5

I have changed both lines and it still will not join. I also tried to access the elasticsearch using the browser but I do not know the password, it never showed it when I tried to set it up initially, as it does when you run setup in a single node, and it wont let me reset it as it does not know the health of the cluster. As I am just trying to set it up and I have no data yet should I try again from scratch and do you know the best way to do it. I am using 8.11.3 the latest release. Or should I just set up the master first stand alone and then add the second node as data. I was not sure that could be done.

6

I'd suggest sharing some more comprehensive logs, it's pretty much impossible to tell what's going on from the selected exerpts you've shared so far.

See also the discovery troubleshooting guide.

7

Thanks again for helping. I deleted the log then ran both again and waited till I got the error and here is the log from the master:

[2024-01-10T13:41:36,066][INFO ][o.a.l.i.v.PanamaVectorizationProvider] [bytelink-1] Java vector incubator API enabled; uses preferredBitSize=128
[2024-01-10T13:41:38,405][INFO ][o.e.n.Node               ] [bytelink-1] version[8.11.3], pid[33332], build[zip/64cf052f3b56b1fd4449f5454cb88aca7e739d9a/2023-12-08T11:33:53.634979452Z], OS[Windows Server 2022/10.0/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/21.0.1/21.0.1+12-29]
[2024-01-10T13:41:38,413][INFO ][o.e.n.Node               ] [bytelink-1] JVM home [D:\Elastic ELK Stack\elasticsearch-8.11.3\jdk], using bundled JDK [true]
[2024-01-10T13:41:38,414][INFO ][o.e.n.Node               ] [bytelink-1] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -Djava.security.manager=allow, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j2.formatMsgNoLookups=true, -Djava.locale.providers=SPI,COMPAT, --add-opens=java.base/java.io=org.elasticsearch.preallocate, -XX:+UseG1GC, -Djava.io.tmpdir=C:\Users\ADMINI~1.BYT\AppData\Local\Temp\elasticsearch, --add-modules=jdk.incubator.vector, -XX:+HeapDumpOnOutOfMemoryError, -XX:+ExitOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,level,pid,tags:filecount=32,filesize=64m, -Xms31744m, -Xmx31744m, -XX:MaxDirectMemorySize=16642998272, -XX:InitiatingHeapOccupancyPercent=30, -XX:G1ReservePercent=25, -Des.distribution.type=zip, --module-path=D:\Elastic ELK Stack\elasticsearch-8.11.3\lib, --add-modules=jdk.net, --add-modules=ALL-MODULE-PATH, -Djdk.module.main=org.elasticsearch.server]
[2024-01-10T13:41:44,853][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [repository-url]
[2024-01-10T13:41:44,855][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [rest-root]
[2024-01-10T13:41:44,856][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-core]
[2024-01-10T13:41:44,856][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-redact]
[2024-01-10T13:41:44,857][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [ingest-user-agent]
[2024-01-10T13:41:44,857][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-async-search]
[2024-01-10T13:41:44,858][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-monitoring]
[2024-01-10T13:41:44,859][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [repository-s3]
[2024-01-10T13:41:44,859][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-analytics]
[2024-01-10T13:41:44,860][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-ent-search]
[2024-01-10T13:41:44,861][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-autoscaling]
[2024-01-10T13:41:44,861][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [lang-painless]
[2024-01-10T13:41:44,861][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-ml]
[2024-01-10T13:41:44,862][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [lang-mustache]
[2024-01-10T13:41:44,863][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [legacy-geo]
[2024-01-10T13:41:44,863][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-ql]
[2024-01-10T13:41:44,864][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [rank-rrf]
[2024-01-10T13:41:44,869][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [analysis-common]
[2024-01-10T13:41:44,870][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [transport-netty4]
[2024-01-10T13:41:44,871][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [aggregations]
[2024-01-10T13:41:44,871][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [ingest-common]
[2024-01-10T13:41:44,872][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [frozen-indices]
[2024-01-10T13:41:44,873][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-identity-provider]
[2024-01-10T13:41:44,873][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-shutdown]
[2024-01-10T13:41:44,874][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-text-structure]
[2024-01-10T13:41:44,875][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [snapshot-repo-test-kit]
[2024-01-10T13:41:44,875][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [ml-package-loader]
[2024-01-10T13:41:44,876][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [kibana]
[2024-01-10T13:41:44,878][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [constant-keyword]
[2024-01-10T13:41:44,879][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-logstash]
[2024-01-10T13:41:44,880][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-ccr]
[2024-01-10T13:41:44,881][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-graph]
[2024-01-10T13:41:44,882][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-esql]
[2024-01-10T13:41:44,883][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [parent-join]
[2024-01-10T13:41:44,885][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-enrich]
[2024-01-10T13:41:44,886][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [repositories-metering-api]
[2024-01-10T13:41:44,889][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [transform]
[2024-01-10T13:41:44,890][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [repository-azure]
[2024-01-10T13:41:44,891][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [repository-gcs]
[2024-01-10T13:41:44,907][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [spatial]
[2024-01-10T13:41:44,910][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [apm]
[2024-01-10T13:41:44,911][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [mapper-extras]
[2024-01-10T13:41:44,912][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [mapper-version]
[2024-01-10T13:41:44,914][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-rollup]
[2024-01-10T13:41:44,915][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [percolator]
[2024-01-10T13:41:44,916][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [data-streams]
[2024-01-10T13:41:44,917][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-stack]
[2024-01-10T13:41:44,918][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [rank-eval]
[2024-01-10T13:41:44,930][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [reindex]
[2024-01-10T13:41:44,934][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-security]
[2024-01-10T13:41:44,937][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [blob-cache]
[2024-01-10T13:41:44,942][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [searchable-snapshots]
[2024-01-10T13:41:44,944][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-slm]
[2024-01-10T13:41:44,951][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [snapshot-based-recoveries]
[2024-01-10T13:41:44,981][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-watcher]
[2024-01-10T13:41:44,987][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [old-lucene-versions]
[2024-01-10T13:41:44,988][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-ilm]
[2024-01-10T13:41:44,990][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-voting-only-node]
[2024-01-10T13:41:44,991][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-deprecation]
[2024-01-10T13:41:44,993][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-fleet]
[2024-01-10T13:41:45,005][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-aggregate-metric]
[2024-01-10T13:41:45,006][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-downsample]
[2024-01-10T13:41:45,008][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-profiling]
[2024-01-10T13:41:45,008][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [ingest-geoip]
[2024-01-10T13:41:45,009][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [inference]
[2024-01-10T13:41:45,010][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-write-load-forecaster]
[2024-01-10T13:41:45,011][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [search-business-rules]
[2024-01-10T13:41:45,013][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [ingest-attachment]
[2024-01-10T13:41:45,014][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [wildcard]
[2024-01-10T13:41:45,016][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [unsigned-long]
[2024-01-10T13:41:45,018][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-sql]
[2024-01-10T13:41:45,019][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [runtime-fields-common]
[2024-01-10T13:41:45,019][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-async]
[2024-01-10T13:41:45,020][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [vector-tile]
[2024-01-10T13:41:45,020][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [lang-expression]
[2024-01-10T13:41:45,021][INFO ][o.e.p.PluginsService     ] [bytelink-1] loaded module [x-pack-eql]
[2024-01-10T13:41:50,517][INFO ][o.e.e.NodeEnvironment    ] [bytelink-1] using [1] data paths, mounts [[System Files (D:)]], net usable_space [329.4gb], net total_space [620.8gb], types [NTFS]
[2024-01-10T13:41:50,518][INFO ][o.e.e.NodeEnvironment    ] [bytelink-1] heap size [31gb], compressed ordinary object pointers [true]
[2024-01-10T13:41:50,660][INFO ][o.e.n.Node               ] [bytelink-1] node name [bytelink-1], node ID [fyv3RzfXQaaIMiz0C_GtnA], cluster name [bytelink-network], roles [data_cold, data, remote_cluster_client, master, data_warm, data_content, transform, data_hot, ml, data_frozen, ingest]
[2024-01-10T13:41:55,542][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [bytelink-1] [controller/26560] [Main.cc@123] controller (64 bit): Version 8.11.3 (Build c16ff912638f0a) Copyright (c) 2023 Elasticsearch BV
[2024-01-10T13:41:56,169][INFO ][o.e.x.s.Security         ] [bytelink-1] Security is enabled
[2024-01-10T13:41:57,277][INFO ][o.e.x.s.a.s.FileRolesStore] [bytelink-1] parsed [0] roles from file [D:\Elastic ELK Stack\elasticsearch-8.11.3\config\roles.yml]
[2024-01-10T13:41:59,012][INFO ][o.e.x.p.ProfilingPlugin  ] [bytelink-1] Profiling is enabled
[2024-01-10T13:41:59,056][INFO ][o.e.x.p.ProfilingPlugin  ] [bytelink-1] profiling index templates will not be installed or reinstalled
[2024-01-10T13:42:00,265][INFO ][o.e.t.n.NettyAllocator   ] [bytelink-1] creating NettyAllocator with the following configs: [name=elasticsearch_configured, chunk_size=1mb, suggested_max_allocation_size=1mb, factors={es.unsafe.use_netty_default_chunk_and_page_size=false, g1gc_enabled=true, g1gc_region_size=16mb}]
[2024-01-10T13:42:00,337][INFO ][o.e.i.r.RecoverySettings ] [bytelink-1] using rate limit [40mb] with [default=40mb, read=0b, write=0b, max=0b]
[2024-01-10T13:42:00,452][INFO ][o.e.d.DiscoveryModule    ] [bytelink-1] using discovery type [multi-node] and seed hosts providers [settings]
[2024-01-10T13:42:03,147][INFO ][o.e.n.Node               ] [bytelink-1] initialized
[2024-01-10T13:42:03,149][INFO ][o.e.n.Node               ] [bytelink-1] starting ...
[2024-01-10T13:42:03,189][INFO ][o.e.x.s.c.f.PersistentCache] [bytelink-1] persistent cache index loaded
[2024-01-10T13:42:03,193][INFO ][o.e.x.d.l.DeprecationIndexingComponent] [bytelink-1] deprecation component started
[2024-01-10T13:42:03,349][INFO ][o.e.t.TransportService   ] [bytelink-1] publish_address {10.30.1.12:9300}, bound_addresses {10.30.1.12:9300}
[2024-01-10T13:42:03,996][INFO ][o.e.b.BootstrapChecks    ] [bytelink-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2024-01-10T13:42:04,116][WARN ][o.e.c.c.ClusterBootstrapService] [bytelink-1] this node is locked into cluster UUID [4f6o1sbvT4KsqqbIgdLb1g] but [cluster.initial_master_nodes] is set to [bytelink-1]; remove this setting to avoid possible data loss caused by subsequent cluster bootstrap attempts; for further information see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/important-settings.html#initial_master_nodes
[2024-01-10T13:42:04,285][INFO ][o.e.c.s.MasterService    ] [bytelink-1] elected-as-master ([1] nodes joined in term 5)[_FINISH_ELECTION_, {bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003} completing election], term: 5, version: 64, delta: master node changed {previous [], current [{bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}]}
[2024-01-10T13:42:04,509][INFO ][o.e.c.s.ClusterApplierService] [bytelink-1] master node changed {previous [], current [{bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}]}, term: 5, version: 64, reason: Publication{term=5, version=64}
[2024-01-10T13:42:04,563][INFO ][o.e.c.f.AbstractFileWatchingService] [bytelink-1] starting file watcher ...
[2024-01-10T13:42:04,588][INFO ][o.e.h.AbstractHttpServerTransport] [bytelink-1] publish_address {10.30.1.12:9200}, bound_addresses {10.30.1.12:9200}
[2024-01-10T13:42:04,590][INFO ][o.e.n.Node               ] [bytelink-1] started {bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{ml.machine_memory=103012937728, transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0, ml.max_jvm_size=33285996544, ml.allocated_processors_double=4.0, ml.allocated_processors=4}
[2024-01-10T13:42:04,589][INFO ][o.e.c.c.NodeJoinExecutor ] [bytelink-1] node-join: [{bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}] with reason [completing election]
[2024-01-10T13:42:04,590][INFO ][o.e.c.f.AbstractFileWatchingService] [bytelink-1] file settings service up and running [tid=69]
[2024-01-10T13:42:05,134][INFO ][o.e.l.ClusterStateLicenseService] [bytelink-1] license [cbeeff90-670e-4fec-bcb3-16cd7d97e8e7] mode [basic] - valid
[2024-01-10T13:42:05,145][INFO ][o.e.x.s.a.Realms         ] [bytelink-1] license mode is [basic], currently licensed security realms are [reserved/reserved,file/default_file,native/default_native]
[2024-01-10T13:42:05,156][INFO ][o.e.g.GatewayService     ] [bytelink-1] recovered [1] indices into cluster_state
[2024-01-10T13:42:05,682][INFO ][o.e.i.m.MapperService    ] [bytelink-1] [.security-7] reloading search analyzers
[2024-01-10T13:42:05,927][INFO ][o.e.h.n.s.HealthNodeTaskExecutor] [bytelink-1] Node [{bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}] is selected as the current health node.
[2024-01-10T13:42:05,967][INFO ][o.e.c.r.a.AllocationService] [bytelink-1] current.health="GREEN" message="Cluster health status changed from [RED] to [GREEN] (reason: [shards started [[.security-7][0]]])." previous.health="RED" reason="shards started [[.security-7][0]]"
[2024-01-10T13:42:30,522][WARN ][o.e.c.c.Coordinator      ] [bytelink-1] received join request from [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{mfsEE89TS8udhyj7bq3pkA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}] but could not connect back to the joining node
org.elasticsearch.transport.ConnectTransportException: [bytelink-2][10.30.1.13:9300] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1156) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:322) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:234) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:62) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$1(SecurityNetty4Transport.java:346) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	... 26 more
Caused by: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	... 26 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
Caused by: java.net.ConnectException: Connection timed out: no further information
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
	... 7 more
[2024-01-10T13:42:52,324][WARN ][o.e.c.c.Coordinator      ] [bytelink-1] received join request from [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{mfsEE89TS8udhyj7bq3pkA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}] but could not connect back to the joining node
org.elasticsearch.transport.ConnectTransportException: [bytelink-2][10.30.1.13:9300] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1156) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:322) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:234) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:62) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$1(SecurityNetty4Transport.java:346) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	... 26 more
Caused by: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	... 26 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
Caused by: java.net.ConnectException: Connection timed out: no further information
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
	... 7 more
[2024-01-10T13:43:13,360][WARN ][o.e.c.c.Coordinator      ] [bytelink-1] received join request from [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{mfsEE89TS8udhyj7bq3pkA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}] but could not connect back to the joining node
org.elasticsearch.transport.ConnectTransportException: [bytelink-2][10.30.1.13:9300] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1156) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:322) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:234) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:62) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$1(SecurityNetty4Transport.java:346) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	... 26 more
Caused by: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	... 26 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
Caused by: java.net.ConnectException: Connection timed out: no further information
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
	... 7 more
[2024-01-10T13:43:31,088][INFO ][o.e.n.Node               ] [bytelink-1] stopping ...
[2024-01-10T13:43:31,092][INFO ][o.e.c.f.AbstractFileWatchingService] [bytelink-1] shutting down watcher thread
[2024-01-10T13:43:31,095][INFO ][o.e.c.f.AbstractFileWatchingService] [bytelink-1] watcher service stopped
[2024-01-10T13:43:31,100][INFO ][o.e.x.w.WatcherService   ] [bytelink-1] stopping watch service, reason [shutdown initiated]
[2024-01-10T13:43:31,100][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [bytelink-1] [controller/26560] [Main.cc@176] ML controller exiting
[2024-01-10T13:43:31,102][INFO ][o.e.x.m.p.NativeController] [bytelink-1] Native controller process has stopped - no new native processes can be started
[2024-01-10T13:43:31,102][INFO ][o.e.x.w.WatcherLifeCycleService] [bytelink-1] watcher has stopped and shutdown
[2024-01-10T13:43:34,404][WARN ][o.e.c.c.Coordinator      ] [bytelink-1] received join request from [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{mfsEE89TS8udhyj7bq3pkA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}] but could not connect back to the joining node
org.elasticsearch.transport.ConnectTransportException: [bytelink-2][10.30.1.13:9300] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1156) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:322) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:234) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:62) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$1(SecurityNetty4Transport.java:346) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	... 26 more
Caused by: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	... 26 more
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
Caused by: java.net.ConnectException: Connection timed out: no further information
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
	... 7 more
[2024-01-10T13:43:34,529][INFO ][o.e.n.Node               ] [bytelink-1] stopped
[2024-01-10T13:43:34,530][INFO ][o.e.n.Node               ] [bytelink-1] closing ...
[2024-01-10T13:43:34,593][INFO ][o.e.n.Node               ] [bytelink-1] closed
8

and here is the tail end of the data node as it was too large for file:

[2024-01-10T13:43:18,291][WARN ][o.e.c.c.JoinHelper       ] [bytelink-2] last failed join attempt was 4.9s ago, failed to join {bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.machine_memory=103012937728, ml.allocated_processors=4, ml.allocated_processors_double=4.0, ml.max_jvm_size=33285996544, ml.config_version=11.0.0} with JoinRequest{sourceNode={bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{mfsEE89TS8udhyj7bq3pkA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}{ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0}, compatibilityVersions=CompatibilityVersions[transportVersion=8512001, systemIndexMappingsVersion={.secrets-inference=MappingsVersion[version=1, hash=-1434574148], .triggered_watches=MappingsVersion[version=1, hash=-502826165], .fleet-agents-7=MappingsVersion[version=1, hash=-644511608], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .ml-config=MappingsVersion[version=1, hash=1455005892], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=1, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-1=MappingsVersion[version=1, hash=1647955624], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-840564854], .fleet-actions-7=MappingsVersion[version=1, hash=961753072], .tasks=MappingsVersion[version=0, hash=-945584329], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .ml-meta=MappingsVersion[version=1, hash=976416370], .security-7=MappingsVersion[version=1, hash=-1061511639], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=2, hash=-1534099917], .inference=MappingsVersion[version=1, hash=645017346], .async-search=MappingsVersion[version=0, hash=-1403744380], .fleet-policies-7=MappingsVersion[version=1, hash=762674907], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], minimumTerm=7391, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [bytelink-1][10.30.1.12:9300][internal:cluster/coordination/join]
Caused by: org.elasticsearch.transport.NodeDisconnectedException: [bytelink-2][10.30.1.13:9300][internal:cluster/coordination/join] failure when opening connection back from [{bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}] to [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{mfsEE89TS8udhyj7bq3pkA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}]
Caused by: org.elasticsearch.transport.ConnectTransportException: [bytelink-2][10.30.1.13:9300] connect_exception
	at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1156) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:322) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:234) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:62) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$1(SecurityNetty4Transport.java:346) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:62) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$1(SecurityNetty4Transport.java:346) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.common.io.stream.NotSerializableExceptionWrapper: execution_exception: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
	at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178) ~[elasticsearch-8.11.3.jar:?]
	at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:62) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$1(SecurityNetty4Transport.java:346) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
	at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:321) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:337) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.io.IOException: Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
Caused by: java.io.IOException: Connection timed out: no further information
	at sun.nio.ch.Net.pollConnect(Native Method) ~[?:?]
	at sun.nio.ch.Net.pollConnectNow(Net.java:682) ~[?:?]
	at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:973) ~[?:?]
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:337) ~[?:?]
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:334) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
	at java.lang.Thread.run(Thread.java:1583) ~[?:?]
[2024-01-10T13:43:18,298][WARN ][o.e.c.c.ClusterFormationFailureHelper] [bytelink-2] master not discovered yet: have discovered [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{mfsEE89TS8udhyj7bq3pkA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}, {bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}]; discovery will continue using [10.30.1.12:9300] from hosts providers and [] from last-known cluster state; node term 7391, last-accepted version 0 in term 0; joining [{bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}] in term [7391] has status [waiting for response] after [4.8s/4814ms]; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.11/discovery-troubleshooting.html
[2024-01-10T13:43:22,878][INFO ][o.e.n.Node               ] [bytelink-2] stopping ...
[2024-01-10T13:43:22,882][INFO ][o.e.x.w.WatcherService   ] [bytelink-2] stopping watch service, reason [shutdown initiated]
[2024-01-10T13:43:22,882][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [bytelink-2] [controller/12180] [Main.cc@176] ML controller exiting
[2024-01-10T13:43:22,884][INFO ][o.e.x.w.WatcherLifeCycleService] [bytelink-2] watcher has stopped and shutdown
[2024-01-10T13:43:22,884][INFO ][o.e.x.m.p.NativeController] [bytelink-2] Native controller process has stopped - no new native processes can be started
[2024-01-10T13:43:23,151][INFO ][o.e.c.c.JoinHelper       ] [bytelink-2] failed to join {bytelink-1}{fyv3RzfXQaaIMiz0C_GtnA}{r4kYctGNThuwA8vwXY62-Q}{bytelink-1}{10.30.1.12}{10.30.1.12:9300}{cdfhilmrstw}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.machine_memory=103012937728, ml.allocated_processors=4, ml.allocated_processors_double=4.0, ml.max_jvm_size=33285996544, ml.config_version=11.0.0} with JoinRequest{sourceNode={bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{mfsEE89TS8udhyj7bq3pkA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}{ml.config_version=11.0.0, xpack.installed=true, transform.config_version=10.0.0}, compatibilityVersions=CompatibilityVersions[transportVersion=8512001, systemIndexMappingsVersion={.secrets-inference=MappingsVersion[version=1, hash=-1434574148], .triggered_watches=MappingsVersion[version=1, hash=-502826165], .fleet-agents-7=MappingsVersion[version=1, hash=-644511608], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .ml-config=MappingsVersion[version=1, hash=1455005892], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=1, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-1=MappingsVersion[version=1, hash=1647955624], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-840564854], .fleet-actions-7=MappingsVersion[version=1, hash=961753072], .tasks=MappingsVersion[version=0, hash=-945584329], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .ml-meta=MappingsVersion[version=1, hash=976416370], .security-7=MappingsVersion[version=1, hash=-1061511639], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=2, hash=-1534099917], .inference=MappingsVersion[version=1, hash=645017346], .async-search=MappingsVersion[version=0, hash=-1403744380], .fleet-policies-7=MappingsVersion[version=1, hash=762674907], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], minimumTerm=7391, optionalJoin=Optional.empty}
org.elasticsearch.transport.NodeDisconnectedException: [bytelink-1][10.30.1.12:9300][internal:cluster/coordination/join] disconnected
[2024-01-10T13:43:23,198][INFO ][o.e.n.Node               ] [bytelink-2] stopped
[2024-01-10T13:43:23,199][INFO ][o.e.n.Node               ] [bytelink-2] closing ...
[2024-01-10T13:43:23,225][INFO ][o.e.n.Node               ] [bytelink-2] closed
9

Great, thanks. Looks like a connectivity issue to me:

[2024-01-10T13:42:30,522][WARN ][o.e.c.c.Coordinator      ] [bytelink-1] received join request from [{bytelink-2}{6lgj7xkVQWOs22q29YKnHA}{mfsEE89TS8udhyj7bq3pkA}{bytelink-2}{10.30.1.13}{10.30.1.13:9300}{di}{8.11.3}{7000099-8500003}{transform.config_version=10.0.0, xpack.installed=true, ml.config_version=11.0.0}]
but could not connect back to the joining node
Caused by: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedConnectException:
Connection timed out: no further information: 10.30.1.13/10.30.1.13:9300
10

Once again thank you. I finally found the problem and you were correct it was a communication issue. I thought I had configured the firewall correctly but found I had down it for the wrong one. Thank you once again. I really appreciate it as I am just learning the process before we adopt it for the company.

1 Like
11

Excellent.

Leandro's earlier comments are right tho, if you only have two nodes we recommend only one of them being master-eligible. See these docs for more information about cluster design and these docs about how to safely remove a master-eligible node.

12

Thankyou for the help and the docs.

13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.