I cannot transport data between two elasticsearch nodes

quanganhhnv · February 20, 2024, 9:57am

I have two machines using elasticSearch, each machine I have the following configuration set

Machine 1

cluster.name: NDC
node.name: node-1
node.roles: [ master, data ]
network.host: 30.30.30.65
http.port: 9200
discovery.seed_hosts: ["30.30.30.65:9200", "30.30.30.15:9200"]
cluster.initial_master_nodes: node-1, node-2

Machine 2

cluster.name: NDC
node.name: node-2
node.roles: [ master, data ]
network.host: 30.30.30.15
http.port: 9200
discovery.seed_hosts: ["30.30.30.65", "30.30.30.15"]
cluster.initial_master_nodes: node-1, node-2

I want to be able to transport data through each other but I can't. Can anyone help me?

Christian_Dahlqvist · February 20, 2024, 10:09am

Elasticsearch nodes communicate over port 9300, not 9200, so you have the wrong port specified here. As 9300 is default you can remove port completely here. Also make sure port 9300 is not blocked by firewall.

quanganhhnv · February 22, 2024, 9:14am

Yes, thank for reply, i was removed port. After that, I could ping each other to port 9300 and 9200, but still couldn't see the index and synchronize each other's data. Am I missing anything?

Christian_Dahlqvist · February 22, 2024, 9:17am

I do not see anything obviously wrong. Are both nodes able to connect to the other on port 9300?

What is in the Elasticsearch logs on the two nodes?

quanganhhnv · February 22, 2024, 9:43am

Yes, sir. Both nodes able to connect to the other on port 9300. This my Elasticsearch logs. It's nothing here because i was started success today

Christian_Dahlqvist · February 22, 2024, 9:49am

It looks like node 2 has previously started up as a standalone cluster and therefore can not join the other node. I would recommend shutting down node 2 and clear its data directory so it can join as a new, empty node.

It is worth noting that having 2 nodes being master eligible is bad practice as it does not give you any resilience or high availability. If any one of the nodes go down your cluster will be inoperable. I would therefore recommend adding one more master eligible node to the cluster. This can be a smaller didicated master node and does not necessarily need to be the same size and specification as the 2 master/data nodes.

quanganhhnv · February 22, 2024, 9:52am

So I will only set 1 node as master, right?

quanganhhnv · February 22, 2024, 9:54am

Okays, thank you, i will try

Christian_Dahlqvist · February 22, 2024, 9:59am

If you only have 1 master eligible node you can handle the dedicated data node going down, but if the master node fails the cluster will still be inoperable and if it can not be restored you may lose the data unless you can restore from a snapshot. It is better to set up a 3-node cluster where all nodes are master eligible.

quanganhhnv · February 22, 2024, 10:16am

Oh, when I was running just now, I encountered these warnings. Can you check for me?

Christian_Dahlqvist · February 22, 2024, 10:17am

Please do not post images of text as they are very hard to read and impossible to search or copy from/quote.

quanganhhnv · February 22, 2024, 10:18am

Yes, is my fall. It's here

[2024-02-22T17:12:17,439][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered or elected yet, an election requires two nodes with ids [WiYq6H9BQe6p-xIl96-54Q, yzFub9mVToeRWEPTmgTrYA], have only discovered non-quorum [{node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}]; discovery will continue using [30.30.30.15:9300] from hosts providers and [{node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 17, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-22T17:12:27,439][WARN ][o.e.n.Node               ] [node-1] timed out while waiting for initial discovery state - timeout: 30s
[2024-02-22T17:12:27,456][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered or elected yet, an election requires two nodes with ids [WiYq6H9BQe6p-xIl96-54Q, yzFub9mVToeRWEPTmgTrYA], have only discovered non-quorum [{node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}]; discovery will continue using [30.30.30.15:9300] from hosts providers and [{node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 17, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-22T17:12:27,489][INFO ][o.e.h.AbstractHttpServerTransport] [node-1] publish_address {30.30.30.65:9200}, bound_addresses {[::]:9200}
[2024-02-22T17:12:27,523][INFO ][o.e.n.Node               ] [node-1] started {node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}
[2024-02-22T17:12:37,473][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered or elected yet, an election requires two nodes with ids [WiYq6H9BQe6p-xIl96-54Q, yzFub9mVToeRWEPTmgTrYA], have only discovered non-quorum [{node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}]; discovery will continue using [30.30.30.15:9300] from hosts providers and [{node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 17, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-22T17:12:53,961][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node-1] master not discovered or elected yet, an election requires two nodes with ids [WiYq6H9BQe6p-xIl96-54Q, yzFub9mVToeRWEPTmgTrYA], have only discovered non-quorum [{node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}]; discovery will continue using [30.30.30.15:9300] from hosts providers and [{node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}] from last-known cluster state; node term 17, last-accepted version 0 in term 0; for troubleshooting guidance, see https://www.elastic.co/guide/en/elasticsearch/reference/8.12/discovery-troubleshooting.html
[2024-02-22T17:13:12,473][WARN ][r.suppressed             ] [node-1] path: /_cat/indices/, params: {}, status: 503org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];
        at org.elasticsearch.server@8.12.0/org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:188)
        at org.elasticsearch.server@8.12.0/org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:71)
        at org.elasticsearch.server@8.12.0/org.elasticsearch.action.admin.indices.stats.TransportIndicesStatsAction.checkGlobalBlock(TransportIndicesStatsAction.java:37)
        at org.elasticsearch.server@8.12.0/org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.doExecuteForked(TransportBroadcastByNodeAction.java:239)
        at org.elasticsearch.server@8.12.0/org.elasticsearch.action.support.broadcast.node.TransportBroadcastByNodeAction.lambda$doExecute$1(TransportBroadcastByNodeAction.java:232)

See logs for more details.

[2024-02-22T17:14:47,818][INFO ][o.e.c.c.JoinHelper       ] [node-1] failed to join {node-2}{yzFub9mVToeRWEPTmgTrYA}{g177CoeTQpiQ5KlAAhCV0g}{node-2}{30.30.30.15}{30.30.30.15:9300}{dm}{8.12.1}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true} with JoinRequest{sourceNode={node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}, compatibilityVersions=CompatibilityVersions[transportVersion=8560000, systemIndexMappingsVersion={.triggered_watches=MappingsVersion[version=1, hash=-502826165], .secrets-inference=MappingsVersion[version=1, hash=-1434574148], .fleet-agents-7=MappingsVersion[version=1, hash=1021268528], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .ml-config=MappingsVersion[version=1, hash=1455005892], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=2, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-1=MappingsVersion[version=1, hash=1647955624], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-840564854], .fleet-actions-7=MappingsVersion[version=1, hash=961753072], .tasks=MappingsVersion[version=0, hash=-945584329], .ml-meta=MappingsVersion[version=1, hash=976416370], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .security-7=MappingsVersion[version=1, hash=-1061511639], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=3, hash=919553140], .inference=MappingsVersion[version=1, hash=645017346], .async-search=MappingsVersion[version=0, hash=-1403744380], .fleet-policies-7=MappingsVersion[version=1, hash=762674907], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], features=[features_supported, usage.data_tiers.precalculate_stats, health.dsl.info], minimumTerm=18, optionalJoin=Optional[Join[votingNode={node-1}{WiYq6H9BQe6p-xIl96-54Q}{5laDMQOiQr2x6eMu-DRWXQ}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}, masterCandidateNode={node-2}{yzFub9mVToeRWEPTmgTrYA}{g177CoeTQpiQ5KlAAhCV0g}{node-2}{30.30.30.15}{30.30.30.15:9300}{dm}{8.12.1}{7000099-8500010}{ml.config_version=12.0.0, transform.config_version=10.0.0, xpack.installed=true}, term=18, lastAcceptedTerm=0, lastAcceptedVersion=0]]}org.elasticsearch.transport.RemoteTransportException: [node-2][30.30.30.15:9300][internal:cluster/coordination/join]

quanganhhnv · February 22, 2024, 10:20am

[2024-02-22T17:37:22,971][INFO ][o.e.c.c.JoinHelper       ] [node-1] failed to join {node-2}{yzFub9mVToeRWEPTmgTrYA}{Zude55ZhR_KIEsVVfVu4wA}{node-2}{30.30.30.15}{30.30.30.15:9300}{dm}{8.12.1}{7000099-8500010}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true} with JoinRequest{sourceNode={node-1}{vt8HhXnXQ3q-q9fEXhld4w}{dEv1A2z4RtOt84O1tAp97g}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}, compatibilityVersions=CompatibilityVersions[transportVersion=8560000, systemIndexMappingsVersion={.secrets-inference=MappingsVersion[version=1, hash=-1434574148], .triggered_watches=MappingsVersion[version=1, hash=-502826165], .fleet-agents-7=MappingsVersion[version=1, hash=1021268528], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .ml-config=MappingsVersion[version=1, hash=1455005892], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=2, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-1=MappingsVersion[version=1, hash=1647955624], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-840564854], .fleet-actions-7=MappingsVersion[version=1, hash=961753072], .tasks=MappingsVersion[version=0, hash=-945584329], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .ml-meta=MappingsVersion[version=1, hash=976416370], .security-7=MappingsVersion[version=1, hash=-1061511639], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=3, hash=919553140], .async-search=MappingsVersion[version=0, hash=-1403744380], .inference=MappingsVersion[version=1, hash=645017346], .fleet-policies-7=MappingsVersion[version=1, hash=762674907], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], features=[usage.data_tiers.precalculate_stats, features_supported, health.dsl.info], minimumTerm=20, optionalJoin=Optional[Join[votingNode={node-1}{vt8HhXnXQ3q-q9fEXhld4w}{dEv1A2z4RtOt84O1tAp97g}{node-1}{30.30.30.65}{30.30.30.65:9300}{dm}{8.12.0}{7000099-8500008}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}, masterCandidateNode={node-2}{yzFub9mVToeRWEPTmgTrYA}{Zude55ZhR_KIEsVVfVu4wA}{node-2}{30.30.30.15}{30.30.30.15:9300}{dm}{8.12.1}{7000099-8500010}{transform.config_version=10.0.0, ml.config_version=12.0.0, xpack.installed=true}, term=20, lastAcceptedTerm=0, lastAcceptedVersion=0]]}
org.elasticsearch.transport.RemoteTransportException: [node-2][30.30.30.15:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: index [.internal.alerts-observability.metrics.alerts-default-000001/BnVpAPejQrW7PeunuaRLsQ] version not supported: 8500010 maximum compatible index version is: 8500008
	at org.elasticsearch.cluster.coordination.NodeJoinExecutor.ensureIndexCompatibility(NodeJoinExecutor.java:365) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.cluster.coordination.NodeJoinExecutor.lambda$addBuiltInJoinValidators$10(NodeJoinExecutor.java:471) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.cluster.coordination.Coordinator.lambda$validateJoinRequest$11(Coordinator.java:691) ~[elasticsearch-8.12.0.jar:?]
	at java.util.ArrayList.forEach(ArrayList.java:1596) ~[?:?]
	at java.util.Collections$UnmodifiableCollection.forEach(Collections.java:1116) ~[?:?]
	at org.elasticsearch.cluster.coordination.Coordinator.lambda$validateJoinRequest$12(Coordinator.java:691) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.ActionListener.completeWith(ActionListener.java:300) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.cluster.coordination.Coordinator.validateJoinRequest(Coordinator.java:684) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.cluster.coordination.Coordinator$2.onResponse(Coordinator.java:637) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.cluster.coordination.Coordinator$2.onResponse(Coordinator.java:634) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:32) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.transport.ClusterConnectionManager.lambda$connectToNodeOrRetry$1(ClusterConnectionManager.java:146) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.support.SubscribableListener$SuccessResult.complete(SubscribableListener.java:310) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:230) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.support.SubscribableListener.onResponse(SubscribableListener.java:173) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.transport.ClusterConnectionManager.lambda$connectToNodeOrRetry$4(ClusterConnectionManager.java:256) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:178) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$RunAfterActionListener.onResponse(ActionListenerImplementations.java:260) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$MappedActionListener.onResponse(ActionListenerImplementations.java:95) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.transport.TransportService.lambda$handshake$7(TransportService.java:602) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.ActionListenerImplementations$DelegatingFailureActionListener.onResponse(ActionListenerImplementations.java:212) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:49) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1425) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:433) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.transport.InboundHandler$2.doRun(InboundHandler.java:390) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:983) ~[elasticsearch-8.12.0.jar:?]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.12.0.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]

quanganhhnv · February 22, 2024, 10:41am

It's seem because of version of java bro ?

system · March 21, 2024, 10:42am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch nodes Elasticsearch	7	820	April 19, 2017
Elasticsearch nodes on same machine unable to find each other Elasticsearch	4	1273	June 22, 2017
Setting up 2 nodes on same machine Elasticsearch	3	1747	March 3, 2019
Form cluster using multiple nodes from different machine Elasticsearch	2	1198	April 11, 2018
Can't get a two-node cluster to work Elasticsearch	9	778	September 28, 2020

I cannot transport data between two elasticsearch nodes

Related topics