Elastic Clustering - Issuw with Node

Elastic Stack Elasticsearch
1

I followed instructions to connect 2 separate physical machines to the same cluster, however I am getting this error on my data node. Note: My master node (which is also a data node) is up and running. All my Elastic instances are running TLS and SSL.

[2019-07-07T14:47:30,454][WARN ][o.e.c.c.ClusterFormationFailureHelper] [node2] master not discovered yet: have discovered ; discovery will continue using [192.168.1.231:9200, 192.168.1.230:9200] from hosts providers and [{node2}{AR0yxK0OQ2is16Tjj_Wwdg}{1wbG0DISSxCkfDuJX7F19w}{192.168.1.230}{192.168.1.230:9300}{ml.machine_memory=8185450496, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 6, last-accepted version 62 in term 6

2

This looks wrong. You have specified a port in your discovery addresses, but you have used the wrong one. It should be the transport port which defaults to 9300. Remove the ports in your discovery config to allow Elasticsearch to use the default port instead.

3

You were right. Thank you for the suggestion. I changed it and now I get this

[2019-07-07T19:48:42,524][INFO ][o.e.c.c.JoinHelper ] [node2] failed to join {node1}{PQkXwtv-S1WNnjcAS4qxcA}{-8telr4xT-a9zNNn4KOp-g}{192.168.1.231}{192.168.1.231:9300}{ml.machine_memory=8185450496, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={node2}{AR0yxK0OQ2is16Tjj_Wwdg}{4B220xgIRdG8-AiP5g1RDQ}{192.168.1.230}{192.168.1.230:9300}{ml.machine_memory=8185450496, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=17, lastAcceptedTerm=6, lastAcceptedVersion=62, sourceNode={node2}{AR0yxK0OQ2is16Tjj_Wwdg}{4B220xgIRdG8-AiP5g1RDQ}{192.168.1.230}{192.168.1.230:9300}{ml.machine_memory=8185450496, xpack.installed=true, ml.max_open_jobs=20}, targetNode={node1}{PQkXwtv-S1WNnjcAS4qxcA}{-8telr4xT-a9zNNn4KOp-g}{192.168.1.231}{192.168.1.231:9300}{ml.machine_memory=8185450496, ml.max_open_jobs=20, xpack.installed=true}}]}
org.elasticsearch.transport.RemoteTransportException: [node1][192.168.1.231:9300][internal:cluster/coordination/join]
Caused by: java.lang.IllegalStateException: failure when sending a validation request to node
at org.elasticsearch.cluster.coordination.Coordinator$3.onFailure(Coordinator.java:500) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.cluster.coordination.JoinHelper$5.handleException(JoinHelper.java:359) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1124) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.transport.TcpTransport.lambda$handleException$24(TcpTransport.java:1001) ~[elasticsearch-7.1.1.jar:7.1.1]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:681) ~[elasticsearch-7.1.1.jar:7.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]
at java.lang.Thread.run(Thread.java:835) [?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [node2][192.168.1.230:9300][internal:cluster/coordination/join/validate]
Caused by: org.elasticsearch.cluster.coordination.CoordinationStateRejectedException: join validation on cluster state with a different cluster uuid X-_9yureSZarznij8hi9xg than local cluster uuid kVATZcn9TtKnPmqpEXO3tg, rejecting

These two machines were independent clusters first setup just to test. I am assuming somewhere it has an issue. How to fix it?

4

This note in the docs describes the situation you're in, and the solution.

5

That helped! Thanks. I have a couple more questions just to make sure I understand this right.

  1. How do I know that the clusters are properly setup and replicated? One is Master/Data and another one is Data
  2. Elastic has replicated, so now would my searches be better? Do I need to install Kibana on both the nodes or Kibana can stay on a single node and still handle searches and visualizations properly?
6

I'm not really sure what you're asking here. If GET _cluster/health reports green then every shard is allocated, if that's what you mean. It's probably best to read the "getting started" section of the manual and open a new thread with any further questions instead of asking lots of different questions in a single thread.

7

Well my question is still related to the same topic. I can check the status on elasticsearch but does it show the second node connected? Or infact how many nodes are in the cluster etc. I dont see anything obvious in Kibana that shows me that info. I can run CURL commands on the elaticsearch cluster to get the current settings but I was not sure if there is a "visual" way to monitor the entire cluster.

8

Yes, the cluster health API shows the number of nodes in the cluster.

Yes, the Stack Monitoring link near the bottom on the right-hand side in Kibana gives you dashboards that include this too.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.