Not able to create a cluster with ES 5.6?

Elastic Stack Elasticsearch
1

Hi Experts ,

I got 2 nodes for testing purpose and I have new installation of ES 5.6 . I am using below elasticsearch.yml on both the machines .

First node

cluster.name: e-2
path.logs: /opt/es/2/es_logs
bootstrap.memory_lock: true
network.host: x.x.x.70
http.port: 20021-20030
discovery.zen.ping.unicast.hosts: ["x.x.x.70","x.x.x.71"]
discovery.zen.minimum_master_nodes: 1

Second node 

 cluster.name: e-2
    path.logs: /opt/es/2/es_logs
    bootstrap.memory_lock: true
    network.host: x.x.x.71
    http.port: 20021-20030
    discovery.zen.ping.unicast.hosts: ["x.x.x.71","x.x.x.70"]
    discovery.zen.minimum_master_nodes: 1

Now When I start node 1 I got couple of errors and then it start node 1 as master , same as with node 2 but independent master, can someone please help me to understand why node 2 is not joining as cluster node ?

Error

[2017-09-18T13:57:17,445][WARN ][o.e.d.z.UnicastZenPing   ] [P5Q_-BV] [1] failed send ping to {#zen_unicast_x.x.x.70_0#}{1MmVMCsNRPqXrhGE3faEqA}{x.x.x.70}{x.x.x.70:9300}
java.lang.IllegalStateException: handshake failed, mismatched cluster name [Cluster [e-1]] - {#zen_unicast_x.x.x.70_0#}{1MmVMCsNRPqXrhGE3faEqA}{x.x.x.70}{x.x.x.70:9300} 
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:407) ~[elasticsearch-5.6.0.jar:5.6.0]
    at org.elasticsearch.transport.TransportService.handshake(TransportService.java:370) ~[elasticsearch-5.6.0.jar:5.6.0]
    at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:400) ~[elasticsearch-5.6.0.jar:5.6.0]
    at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:507) [elasticsearch-5.6.0.jar:5.6.0]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.6.0.jar:5.6.0]
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.6.0.jar:5.6.0]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_71]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_71]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]

Please help me to understand what I am missing here?

Regards
VG

2

Hi Vikas

java.lang.IllegalStateException: handshake failed, mismatched cluster name [Cluster [e-1]] - {#zen_unicast_x.x.x.70_0#}{1MmVMCsNRPqXrhGE3faEqA}{x.x.x.70}{x.x.x.70:9300}

nodes belonging to the same cluster need to have same cluster name

The log seems to say that one of the nodes has set cluster.name:e-1 ?

Also

same as with node 2 but independent master

not sure about what independent means here, each instance of ES can be configured to act as a master node (node.master:true); when more masters join the same cluster only one of them at any time will be the currently elected master and be the sole actor allowed to make changes to the cluster state.

I hope this helps

3

Thanks Nellicus for quick response

Actually I have another cluster on the same 2 machines that e-1 running on different ports , and that is on ES 5.5.0 . That clsuter is working fine . Not sure why this new cluster is trying to connect to that one .

Here is my Second cluster which is e-1

cluster.name: e-1
path.logs: /opt/es/1/es_logs
bootstrap.memory_lock: true
network.host: x.x.x.70
http.port: 20011-20020
discovery.zen.ping.unicast.hosts: ["x.x.x.71","x.x.x.70"]
discovery.zen.minimum_master_nodes: 2
4

Does this means I cannot run 2 different clusters on the same machines ? So my plan was to test with older and newer ES versions .

5

you can have mutliple clusters on same machines, although this usually makes sense for advanced users only in some very specific scenarios like using servers with huge amount of RAM. Do you really need more clusters to begin with?

Said that, seeing that exception my guess is that you're having nodes from different clusters (e-1,e-2) trying to talk between each other.

you only want nodes of cluster e-1 to talk to other nodes of cluster e-1 only, not e-2.

I suggest going through a review of your
transport protocol and unicast settings to achieve the desired behaviour.

Cheers

6

running on different ports

http.port: 20011-20020

ES nodes use Transport Protocol to form a cluster, not HTTP, so your nodes are probably all running on the default transport ports 9300-...

7

Thanks Nellicus,

You saved my day , your last comment resolve the problem . I need to use Transport protocol things are working now .

May I ask Another query here , since I have only 2 nodes(servers), so what would be the discovery.zen.minimum_master_nodes in this case . As per document i can calculate as (number of master-eligible nodes / 2) + 1 . Right now I am mentioning it as 1. My confusion here is number of master-eligible nodes , is it overall nodes ? if it is then as per calculation (2/2)+1 = 2 . Please correct me if this is wrong understanding .Also does this mean I only have 1 master node in my cluster and if this goes down I will be in trouble or second node will start acting as master as I have set master: true on both the nodes .

As per my understanding 3 node cluster is recommended to avoid split brain problem ?

Cheers
Vikas

8

With only 2 nodes in the cluster, minimum_master_nodes should be set to 2 if they are both master eligible. This means that the cluster will not take writes if one of the nodes is down, which is by design in order to avoid data loss. With 3 master eligible nodes in the cluster, 2 of the bodes is sufficient for quorum/majority, so the cluster can continue operating even with one node missing.

9

Perfect explanation , Thanks Christian

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.