Elasticsearch 7 cluster setup with 3 nodes ---

Elastic Stack Elasticsearch
1

hi,

i tried to setup elasticsearch 7 setup with 3 nodes in rhel7.

my config files: node1

cluster.name: master-node
node.name: elk-tst1
node.master: true
node.data: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.136.142
http.port: 9200
discovery.seed_hosts: ["192.168.136.142", "192.168.136.143", "192.168.136.130"]
cluster.initial_master_nodes: ["192.168.136.142", "192.168.136.143", "192.168.136.130"]
xpack.security.enabled: false

similar configs for node2 &3:

cluster.name: master-node
node.name: elk-tst2
node.master: false
node.data: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.136.143
http.port: 9200
discovery.seed_hosts: ["192.168.136.142", "192.168.136.143", "192.168.136.130"]
cluster.initial_master_nodes: ["192.168.136.142", "192.168.136.143", "192.168.136.130"]
xpack.security.enabled: false

when i do curl below shows:

curl -X GET http://192.168.136.143:9200/_cluster/health?pretty

{
"error" : {
"root_cause" : [
{
"type" : "master_not_discovered_exception",
"reason" : null
}
],
"type" : "master_not_discovered_exception",
"reason" : null
},
"status" : 503
}

here is my logs: /var/log/elasticsearch/master-node.log

[2019-04-16T23:42:49,260][WARN ][o.e.c.c.Coordinator ] [elk-tst1] failed to validate incoming join request from node [{elk-tst2}{w4Faw4s9Tsi1jaZ1Uf3F7A}{Nf6m4Di7StKReCQwmgeIaQ}{192.168.136.143}{192.168.136.143:9300}{ml.machine_memory=1907937280, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [elk-tst2][192.168.136.143:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid LcitTe7KQFCEF8Ww_MuU_g than local cluster uuid or6tpJqPRqSzOdea_eYA7g, rejecting
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
[2019-04-16T23:42:49,982][WARN ][o.e.c.c.Coordinator ] [elk-tst1] failed to validate incoming join request from node [{elk-tst3}{THjWQ6KBTRmrvu1tAw7IYQ}{Jj-dBpYBTtuLi0pQ22XRsQ}{192.168.136.130}{192.168.136.130:9300}{ml.machine_memory=1907937280, ml.max_open_jobs=20, xpack.installed=true}]
org.elasticsearch.transport.RemoteTransportException: [elk-tst3][192.168.136.130:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid LcitTe7KQFCEF8Ww_MuU_g than local cluster uuid sTdyHyX8SFSN5Mr5XH1E0A, rejecting
at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$4(JoinHelper.java:147) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1077) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:751) ~[elasticsearch-7.0.0.jar:7.0.0]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.0.0.jar:7.0.0]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]

can help me , what i did wrong on my configuration ....

2

As you have listed the nodes as seed nodes I believe they need to be master eligible. You should also have at least one data node in the cluster in order to do something with it.

3

i pasted wrong info of node2 &3 config

cluster.name: master-node
node.name: elk-tst2
node.master: false
node.data: true
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 192.168.136.143
http.port: 9200
discovery.seed_hosts: ["192.168.136.142", "192.168.136.143", "192.168.136.130"]
cluster.initial_master_nodes: ["192.168.136.142", "192.168.136.143", "192.168.136.130"]
xpack.security.enabled: false

4

This indicates that these nodes used to be part of different clusters, but are now trying to join into a single cluster. This risks data loss, so is forbidden.

Is this a brand-new cluster? If so, wipe the data paths and start again. See these docs for more information.

2 Likes
5

I believe you should have node.master set to true for all these nodes.

6

Hi ,

@ David i wipe data paths (/var/lib/elasticsearch/nodes)

@Christian_Dahlqvist i set node.master: true , node.data: true for all the 3 nodes.

clustering works fine... tested should be ok...

curl http://192.168.136.143:9200/_cat/nodes?v

ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
192.168.136.142 9 91 3 1.49 0.94 0.38 mdi - elk-tst1
192.168.136.143 12 97 1 0.26 0.27 0.33 mdi * elk-tst2
192.168.136.130 7 93 0 0.03 0.38 0.25 mdi - elk-tst3

curl http://192.168.136.142:9200/_cat/master

AoCcCqrdQ4izXJv-EG8lKg 192.168.136.143 192.168.136.143 elk-tst2

thanks ... :smile:

3 Likes
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.