Problems working with Cluster

khaiolopes · July 1, 2019, 3:17pm

Guys, I'm having trouble working in Cluster with Elastic. I'm a beginner.

I have some outputs in my logs, I've already looked into the solution but so far none has come up in my solution.

Below the configuration of my NODES.

NODE01-Cluster

cluster.name: WeON
node.name: elasticnode-01
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.30.0.3
discovery.zen.ping.unicast.hosts: ["172.30.0.3", "172.30.0.30"]
discovery.zen.minimum_master_nodes: 2

NODE02

cluster.name: WeON
node.name: elasticnode-02
node.master: false
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.30.0.30
discovery.zen.ping.unicast.hosts: ["172.30.0.3", "172.30.0.30"]
discovery.zen.minimum_master_nodes: 2

In both the NODE-Cluster and NODE02 the Elastic service launches successfully but exception are displayed.

LOG NODE01-Cluster

[2019-07-01T12:15:24,115][WARN ][o.e.c.c.Coordinator      ] [elasticnode-01] failed to validate incoming join request from node [{elasticnode-02}{IzvIV7ImQ-WgCcRPfOvyYw}{-Do6vZxeQW-Yj211Im8DSQ}{172.30.0.30}{172.30.0.30:9300}{ml.machine_memory=10489212928, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [elasticnode-02][172.30.0.30:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid 3fMkprhIQBa0n6OBQwCjmA than local cluster uuid ZRFm66GfQyOwkpRPgjzaXA, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.1.jar:7.0.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

LOG NODE02

[2019-07-01T12:16:31,176][INFO ][o.e.c.c.JoinHelper       ] [elasticnode-02] last failed join attempt was 942ms ms ago, failed to join {elasticnode-01}{zgRHXlHARq6Pv1yapez00g}{rtMEDdnyQW2ogs5CXarzug}{172.30.0.3}{172.30.0.3:9300}{ml.machine_memory=10489212928, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticnode-02}{IzvIV7ImQ-WgCcRPfOvyYw}{-Do6vZxeQW-Yj211Im8DSQ}{172.30.0.30}{172.30.0.30:9300}{ml.machine_memory=10489212928, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [elasticnode-01][172.30.0.3:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
        at org.elasticsearch.cluster.coordination.Coordinator$3.onFailure(Coordinator.java:500) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:359) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.0.1.jar:7.0.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.transport.RemoteTransportException: [elasticnode-02][172.30.0.30:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid 3fMkprhIQBa0n6OBQwCjmA than local cluster uuid ZRFm66GfQyOwkpRPgjzaXA, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]

How should I treat this case?

Thanks!

khaiolopes · July 1, 2019, 6:04pm

Can someone help me? Thanks!

dadoonet · July 1, 2019, 6:25pm

Read this and specifically the "Also be patient" part.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

elasticforme · July 1, 2019, 6:30pm

I think version you are using does not support
discover.zen.ping stuff.

also you need "cluster.initial_master_nodes" first time when you start your cluster

zen thing is replaced by
discovery.seed_hosts:

DavidTurner · July 1, 2019, 6:48pm

Also this note in the docs might describe the situation you're in. If so, it also describes the solution.

khaiolopes · July 2, 2019, 6:49pm

Thanks, with the following content achieved some paritcu- larities and set the environment for a production scenario.

Below the settings I applied in my NODE01-Cluster and NODE02.

NODE01-Cluster

cluster.name: clusterelastic
node.name: nodeelastic
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts:
- 10.95.94.104
- 10.95.94.107
cluster.initial_master_nodes:
- 10.95.94.104
- 10.95.94.107
xpack.monitoring.collection.enabled: true

NODE02

cluster.name: clusterelastic
node.name: nodeelastic1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts:
        - 10.95.94.104
        - 10.95.94.107
cluster.initial_master_nodes:
        - 10.95.94.104
        - 10.95.94.107
xpack.monitoring.collection.enabled: true
node.master: false
node.data: true
xpack.security.enabled: false

I also faced the following situation:

[2019-07-02T14:53:27,428][INFO ][o.e.c.c.JoinHelper       ] [nodeelastic1] failed to join {nodeelastic}{IwUPOzKrTO-C3ZBW5MrT0A}{6dabNyDuRDCOQ12PlQJXig}{10.95.94.104}{10.95.94.104:9300}{ml.machine_memory=4147879936, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={nodeelastic1}{90YnMAqoQHqcCbuxk1wB1A}{YEa7vlcATUunl_aJcQC3gA}{10.95.94.107}{10.95.94.107:9300}{ml.machine_memory=4147892224, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=4, lastAcceptedTerm=3, lastAcceptedVersion=24, sourceNode={nodeelastic1}{90YnMAqoQHqcCbuxk1wB1A}{YEa7vlcATUunl_aJcQC3gA}{10.95.94.107}{10.95.94.107:9300}{ml.machine_memory=4147892224, xpack.installed=true, ml.max_open_jobs=20}, targetNode={nodeelastic}{IwUPOzKrTO-C3ZBW5MrT0A}{6dabNyDuRDCOQ12PlQJXig}{10.95.94.104}{10.95.94.104:9300}{ml.machine_memory=4147879936, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [nodeelastic][10.95.94.104:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
        at org.elasticsearch.cluster.coordination.Coordinator$3.onFailure(Coordinator.java:500) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:359) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [nodeelastic1][10.95.94.107:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid -Ss6ga-GRt6sWbcJduoNjA than local cluster uuid f9rT1R9GQlyPCCWqFFPW6w, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) ~[?:?]

With a little research I found the following information:

@ David i wipe data paths (/var/lib/elasticsearch/nodes)

Reference: Elasticsearch 7 cluster setup with 3 nodes --- - #6 by hari.v

To resolve this, I just removed the /var/lib/elasticsearch/nodes directory information with the command:

rm -rf *

After that I have successfully started my Cluster and my NODES integrated successfully.

Thanks for the help and in the next topics I will be more cautious.

system · July 30, 2019, 6:49pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Cluster error Elasticsearch	11	1780	March 8, 2018
Elastic search clustering error Elasticsearch	4	371	June 11, 2018
Unable to setup elasticsearch cluster Elasticsearch	3	1231	May 26, 2017
Create Elasticsearch Cluster (newbie) Elasticsearch	12	3665	April 3, 2018
Data nodes are not able to join master, failed to send join request to master Elasticsearch	2	871	February 25, 2019

Problems working with Cluster

NODE01-Cluster

NODE02

Related topics