Problems working with Cluster

Elastic Stack Elasticsearch
1

Guys, I'm having trouble working in Cluster with Elastic. I'm a beginner.

I have some outputs in my logs, I've already looked into the solution but so far none has come up in my solution.

Below the configuration of my NODES.

NODE01-Cluster

cluster.name: WeON
node.name: elasticnode-01
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.30.0.3
discovery.zen.ping.unicast.hosts: ["172.30.0.3", "172.30.0.30"]
discovery.zen.minimum_master_nodes: 2

NODE02

cluster.name: WeON
node.name: elasticnode-02
node.master: false
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 172.30.0.30
discovery.zen.ping.unicast.hosts: ["172.30.0.3", "172.30.0.30"]
discovery.zen.minimum_master_nodes: 2

In both the NODE-Cluster and NODE02 the Elastic service launches successfully but exception are displayed.

LOG NODE01-Cluster

[2019-07-01T12:15:24,115][WARN ][o.e.c.c.Coordinator      ] [elasticnode-01] failed to validate incoming join request from node [{elasticnode-02}{IzvIV7ImQ-WgCcRPfOvyYw}{-Do6vZxeQW-Yj211Im8DSQ}{172.30.0.30}{172.30.0.30:9300}{ml.machine_memory=10489212928, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [elasticnode-02][172.30.0.30:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid 3fMkprhIQBa0n6OBQwCjmA than local cluster uuid ZRFm66GfQyOwkpRPgjzaXA, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.1.jar:7.0.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]

LOG NODE02

[2019-07-01T12:16:31,176][INFO ][o.e.c.c.JoinHelper       ] [elasticnode-02] last failed join attempt was 942ms ms ago, failed to join {elasticnode-01}{zgRHXlHARq6Pv1yapez00g}{rtMEDdnyQW2ogs5CXarzug}{172.30.0.3}{172.30.0.3:9300}{ml.machine_memory=10489212928, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={elasticnode-02}{IzvIV7ImQ-WgCcRPfOvyYw}{-Do6vZxeQW-Yj211Im8DSQ}{172.30.0.30}{172.30.0.30:9300}{ml.machine_memory=10489212928, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional.empty}
org.elasticsearch.transport.RemoteTransportException: [elasticnode-01][172.30.0.3:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
        at org.elasticsearch.cluster.coordination.Coordinator$3.onFailure(Coordinator.java:500) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:359) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.0.1.jar:7.0.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_111]
Caused by: org.elasticsearch.transport.RemoteTransportException: [elasticnode-02][172.30.0.30:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid 3fMkprhIQBa0n6OBQwCjmA than local cluster uuid ZRFm66GfQyOwkpRPgjzaXA, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:251) ~[?:?]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:309) ~[?:?]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) [elasticsearch-7.0.1.jar:7.0.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.0.1.jar:7.0.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) ~[?:1.8.0_111]

How should I treat this case?

Thanks!

2

Can someone help me? Thanks!

3

Read this and specifically the "Also be patient" part.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

4

I think version you are using does not support
discover.zen.ping stuff.

also you need "cluster.initial_master_nodes" first time when you start your cluster

zen thing is replaced by
discovery.seed_hosts:

5

Also this note in the docs might describe the situation you're in. If so, it also describes the solution.

6

Thanks, with the following content achieved some paritcu- larities and set the environment for a production scenario.

Below the settings I applied in my NODE01-Cluster and NODE02.

NODE01-Cluster

cluster.name: clusterelastic
node.name: nodeelastic
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts:
- 10.95.94.104
- 10.95.94.107
cluster.initial_master_nodes:
- 10.95.94.104
- 10.95.94.107
xpack.monitoring.collection.enabled: true

NODE02

cluster.name: clusterelastic
node.name: nodeelastic1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts:
        - 10.95.94.104
        - 10.95.94.107
cluster.initial_master_nodes:
        - 10.95.94.104
        - 10.95.94.107
xpack.monitoring.collection.enabled: true
node.master: false
node.data: true
xpack.security.enabled: false

I also faced the following situation:

[2019-07-02T14:53:27,428][INFO ][o.e.c.c.JoinHelper       ] [nodeelastic1] failed to join {nodeelastic}{IwUPOzKrTO-C3ZBW5MrT0A}{6dabNyDuRDCOQ12PlQJXig}{10.95.94.104}{10.95.94.104:9300}{ml.machine_memory=4147879936, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={nodeelastic1}{90YnMAqoQHqcCbuxk1wB1A}{YEa7vlcATUunl_aJcQC3gA}{10.95.94.107}{10.95.94.107:9300}{ml.machine_memory=4147892224, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=4, lastAcceptedTerm=3, lastAcceptedVersion=24, sourceNode={nodeelastic1}{90YnMAqoQHqcCbuxk1wB1A}{YEa7vlcATUunl_aJcQC3gA}{10.95.94.107}{10.95.94.107:9300}{ml.machine_memory=4147892224, xpack.installed=true, ml.max_open_jobs=20}, targetNode={nodeelastic}{IwUPOzKrTO-C3ZBW5MrT0A}{6dabNyDuRDCOQ12PlQJXig}{10.95.94.104}{10.95.94.104:9300}{ml.machine_memory=4147879936, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [nodeelastic][10.95.94.104:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
        at org.elasticsearch.cluster.coordination.Coordinator$3.onFailure(Coordinator.java:500) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:359) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [nodeelastic1][10.95.94.107:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid -Ss6ga-GRt6sWbcJduoNjA than local cluster uuid f9rT1R9GQlyPCCWqFFPW6w, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) ~[?:?]

With a little research I found the following information:

@ David i wipe data paths (/var/lib/elasticsearch/nodes)

Reference: Elasticsearch 7 cluster setup with 3 nodes --- - #6 by hari.v

To resolve this, I just removed the /var/lib/elasticsearch/nodes directory information with the command:

rm -rf *

After that I have successfully started my Cluster and my NODES integrated successfully.

Thanks for the help and in the next topics I will be more cautious.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.