Problems working with Cluster

Elastic Stack Elasticsearch
6

Thanks, with the following content achieved some paritcu- larities and set the environment for a production scenario.

Below the settings I applied in my NODE01-Cluster and NODE02.

NODE01-Cluster

cluster.name: clusterelastic
node.name: nodeelastic
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts:
- 10.95.94.104
- 10.95.94.107
cluster.initial_master_nodes:
- 10.95.94.104
- 10.95.94.107
xpack.monitoring.collection.enabled: true

NODE02

cluster.name: clusterelastic
node.name: nodeelastic1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts:
        - 10.95.94.104
        - 10.95.94.107
cluster.initial_master_nodes:
        - 10.95.94.104
        - 10.95.94.107
xpack.monitoring.collection.enabled: true
node.master: false
node.data: true
xpack.security.enabled: false

I also faced the following situation:

[2019-07-02T14:53:27,428][INFO ][o.e.c.c.JoinHelper       ] [nodeelastic1] failed to join {nodeelastic}{IwUPOzKrTO-C3ZBW5MrT0A}{6dabNyDuRDCOQ12PlQJXig}{10.95.94.104}{10.95.94.104:9300}{ml.machine_memory=4147879936, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={nodeelastic1}{90YnMAqoQHqcCbuxk1wB1A}{YEa7vlcATUunl_aJcQC3gA}{10.95.94.107}{10.95.94.107:9300}{ml.machine_memory=4147892224, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=4, lastAcceptedTerm=3, lastAcceptedVersion=24, sourceNode={nodeelastic1}{90YnMAqoQHqcCbuxk1wB1A}{YEa7vlcATUunl_aJcQC3gA}{10.95.94.107}{10.95.94.107:9300}{ml.machine_memory=4147892224, xpack.installed=true, ml.max_open_jobs=20}, targetNode={nodeelastic}{IwUPOzKrTO-C3ZBW5MrT0A}{6dabNyDuRDCOQ12PlQJXig}{10.95.94.104}{10.95.94.104:9300}{ml.machine_memory=4147879936, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [nodeelastic][10.95.94.104:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
        at org.elasticsearch.cluster.coordination.Coordinator$3.onFailure(Coordinator.java:500) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:359) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1111) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.InboundHandler.lambda$handleException$2(InboundHandler.java:246) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:688) ~[elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [nodeelastic1][10.95.94.107:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid -Ss6ga-GRt6sWbcJduoNjA than local cluster uuid f9rT1R9GQlyPCCWqFFPW6w, rejecting
        at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:267) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
        at java.lang.Thread.run(Thread.java:835) ~[?:?]

With a little research I found the following information:

@ David i wipe data paths (/var/lib/elasticsearch/nodes)

Reference: Elasticsearch 7 cluster setup with 3 nodes --- - #6 by hari.v

To resolve this, I just removed the /var/lib/elasticsearch/nodes directory information with the command:

rm -rf *

After that I have successfully started my Cluster and my NODES integrated successfully.

Thanks for the help and in the next topics I will be more cautious.