Minimum cluster config on the same machine for test

Elastic Stack Elasticsearch
1

hi,
I tried to explore the cluster config and I can't configure the minimum config to run 2 nodes on the same machine (only for testing of course !)

so, what is the minimum config for cluster config (ES 5)

elasticsearch1.yml :
cluster.name: cluster-es
node.name: node-1
network.host: 192.168.1.10
http.port: 9200
discovery.zen.minimum_master_nodes: 2

elasticsearch2.yml :
cluster.name: cluster-es
node.name: node-2
network.host: 192.168.1.10
http.port: 9202
discovery.zen.minimum_master_nodes: 2

when I start my 2 nodes, here the log :

2019-04-24T10:58:29,058][WARN ][o.e.d.z.ZenDiscovery ] [node-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{b-BnNdqlTc2aoExHinDR9w}{ueAwxiKkTRGv5shQVuLobw}{192.168.1.10}{192.168.1.10:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-04-24T10:58:32,060][WARN ][o.e.d.z.ZenDiscovery ] [node-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{b-BnNdqlTc2aoExHinDR9w}{ueAwxiKkTRGv5shQVuLobw}{192.168.1.10}{192.168.1.10:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again

It doesn't find the node-2 in the ping processing...

2

You need to configure discovery.zen.ping.unicast.hosts so that each node can find the other:

discovery.zen.ping.unicast.hosts: [192.168.1.10:9300, 192.168.1.10:9302]
3

Thanks, I tried but : failed to send join request to master [{node-2}

Why do you put the port 9300 and 9302 and not 9300 and 9301 ? I tried with 9302 and 9301 but the same error :

[2019-04-25T10:33:59,162][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{b-BnNdqlTc2aoExHinDR9w}{uB2FVblETMuhH2srILlO8g}{192.168.1.10}{192.168.1.10:9301}], reason [RemoteTransportException[[node-2][192.168.1.10:9301][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node-2}{b-BnNdqlTc2aoExHinDR9w}{uB

4

Sorry, I misread your original post. You're setting http.port to 9200 and 9202, but not setting transport.tcp.port, so these nodes will have unpredictable transport ports. It's probably best to set transport.tcp.port on these nodes to be sure what their ports are.

That's very much not the same error :slight_smile: In fact it's not necessarily an error at all (it's only an INFO message).

5

ok for INFO message, the 2 nodes start and I have only the info message "failed to send join request to master"

Now, my http://192.168.1.10:9200/_cat/indices return error :

{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}

my new config :
cluster.name: es-cluster
node.name: node-1
network.host: 192.168.1.10
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["192.168.1.10:9300","192.168.1.10:9302"]
discovery.zen.minimum_master_nodes: 2

I wonder if node.master is to set ? (in ES 5.6)

6

The default for node.master is true so that's ok.

Can you share the logs from both nodes, showing all messages from when you start the node onwards? The information we need should be in there.

7

node1 : elasticsearch.yml
cluster.name: cluster-es
node.name: node-1
network.host: 192.168.1.10
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["192.168.1.10:9300","192.168.1.10:9302"]
discovery.zen.minimum_master_nodes: 2

node2 : elasticsearch.yml
cluster.name: cluster-es
node.name: node-2
network.host: 192.168.1.10
http.port: 9202
transport.tcp.port: 9302
discovery.zen.ping.unicast.hosts: ["192.168.1.10:9300","192.168.1.10:9302"]
discovery.zen.minimum_master_nodes: 2

NODE1 :

[2019-04-26T13:45:10,366][INFO ][o.e.n.Node ] [node-1] initializing ...
[2019-04-26T13:45:10,526][INFO ][o.e.e.NodeEnvironment ] [node-1] using [1] data paths, mounts [[/var (/dev/mapper/vgroot-lvvar)]], net usable_space [10.4gb], net total_space [38.2gb], spins? [possibly], types [ext4]
[2019-04-26T13:45:10,526][INFO ][o.e.e.NodeEnvironment ] [node-1] heap size [1015.6mb], compressed ordinary object pointers [true]
[2019-04-26T13:45:10,581][INFO ][o.e.n.Node ] [node-1] node name [node-1], node ID [b-BnNdqlTc2aoExHinDR9w]
[2019-04-26T13:45:10,581][INFO ][o.e.n.Node ] [node-1] version[5.6.11], pid[15751], build[bc3eef4/2018-08-16T15:25:17.293Z], OS[Linux/3.10.0-327.18.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2019-04-26T13:45:10,581][INFO ][o.e.n.Node ] [node-1] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2019-04-26T13:45:12,458][INFO ][o.e.p.PluginsService ] [node-1] loaded module [aggs-matrix-stats]
[2019-04-26T13:45:12,458][INFO ][o.e.p.PluginsService ] [node-1] loaded module [ingest-common]
[2019-04-26T13:45:12,458][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-expression]
[2019-04-26T13:45:12,459][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-groovy]
[2019-04-26T13:45:12,459][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-mustache]
[2019-04-26T13:45:12,459][INFO ][o.e.p.PluginsService ] [node-1] loaded module [lang-painless]
[2019-04-26T13:45:12,459][INFO ][o.e.p.PluginsService ] [node-1] loaded module [parent-join]
[2019-04-26T13:45:12,459][INFO ][o.e.p.PluginsService ] [node-1] loaded module [percolator]
[2019-04-26T13:45:12,459][INFO ][o.e.p.PluginsService ] [node-1] loaded module [reindex]
[2019-04-26T13:45:12,459][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty3]
[2019-04-26T13:45:12,459][INFO ][o.e.p.PluginsService ] [node-1] loaded module [transport-netty4]
[2019-04-26T13:45:12,459][INFO ][o.e.p.PluginsService ] [node-1] no plugins loaded
[2019-04-26T13:45:16,963][INFO ][o.e.d.DiscoveryModule ] [node-1] using discovery type [zen]
[2019-04-26T13:45:18,915][INFO ][o.e.n.Node ] [node-1] initialized
[2019-04-26T13:45:18,915][INFO ][o.e.n.Node ] [node-1] starting ...
[2019-04-26T13:45:19,599][INFO ][o.e.t.TransportService ] [node-1] publish_address {192.168.1.10:9300}, bound_addresses {192.168.1.10:9300}
[2019-04-26T13:45:19,608][INFO ][o.e.b.BootstrapChecks ] [node-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-04-26T13:45:22,919][WARN ][o.e.d.z.ZenDiscovery ] [node-1] not enough master nodes discovered during pinging (found [[Candidate{node={node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}, clusterStateVersion=-1}]], but needed [2]), pinging again
[2019-04-26T13:45:28,306][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{b-BnNdqlTc2aoExHinDR9w}{jx_HvawBR1-X5WxKbCrtsg}{192.168.1.10}{192.168.1.10:9302}], reason [RemoteTransportException[[node-2][192.168.1.10:9302][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node-2}{b-BnNdqlTc2aoExHinDR9w}{jx_HvawBR1-X5WxKbCrtsg}{192.168.1.10}{192.168.1.10:9302}] not master for join request]; ], tried [3] times
[2019-04-26T13:45:37,532][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{b-BnNdqlTc2aoExHinDR9w}{jx_HvawBR1-X5WxKbCrtsg}{192.168.1.10}{192.168.1.10:9302}], reason [RemoteTransportException[[node-2][192.168.1.10:9302][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node-2}{b-BnNdqlTc2aoExHinDR9w}{jx_HvawBR1-X5WxKbCrtsg}{192.168.1.10}{192.168.1.10:9302}] not master for join request]; ], tried [3] times
[2019-04-26T13:45:43,749][INFO ][o.e.d.z.ZenDiscovery ] [node-1] failed to send join request to master [{node-2}{b-BnNdqlTc2aoExHinDR9w}{jx_HvawBR1-X5WxKbCrtsg}{192.168.1.10}{192.168.1.10:9302}], reason [RemoteTransportException[[node-2][192.168.1.10:9302][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node-2}{b-BnNdqlTc2aoExHinDR9w}{jx_HvawBR1-X5WxKbCrtsg}{192.168.1.10}{192.168.1.10:9302}] not master for join request]; ], tried [3] times
[2019-04-26T13:45:49,803][WARN ][o.e.n.Node ] [node-1] timed out while waiting for initial discovery state - timeout: 30s
[2019-04-26T13:45:49,810][INFO ][o.e.h.n.Netty4HttpServerTransport] [node-1] publish_address {192.168.1.10:9200}, bound_addresses {192.168.1.10:9200}
[2019-04-26T13:45:49,810][INFO ][o.e.n.Node ] [node-1] started

8

NODE2 :

[2019-04-26T13:45:17,866][INFO ][o.e.n.Node ] [node-2] initializing ...
[2019-04-26T13:45:18,225][INFO ][o.e.e.NodeEnvironment ] [node-2] using [1] data paths, mounts [[/var (/dev/mapper/vgroot-lvvar)]], net usable_space [10.4gb], net total_space [38.2gb], spins? [possibly], types [ext4]
[2019-04-26T13:45:18,226][INFO ][o.e.e.NodeEnvironment ] [node-2] heap size [1015.6mb], compressed ordinary object pointers [true]
[2019-04-26T13:45:18,343][INFO ][o.e.n.Node ] [node-2] node name [node-2], node ID [b-BnNdqlTc2aoExHinDR9w]
[2019-04-26T13:45:18,343][INFO ][o.e.n.Node ] [node-2] version[5.6.11], pid[15810], build[bc3eef4/2018-08-16T15:25:17.293Z], OS[Linux/3.10.0-327.18.2.el7.x86_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_181/25.181-b13]
[2019-04-26T13:45:18,343][INFO ][o.e.n.Node ] [node-2] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch2]
[2019-04-26T13:45:20,742][INFO ][o.e.p.PluginsService ] [node-2] loaded module [aggs-matrix-stats]
[2019-04-26T13:45:20,742][INFO ][o.e.p.PluginsService ] [node-2] loaded module [ingest-common]
[2019-04-26T13:45:20,742][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-expression]
[2019-04-26T13:45:20,742][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-groovy]
[2019-04-26T13:45:20,742][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-mustache]
[2019-04-26T13:45:20,742][INFO ][o.e.p.PluginsService ] [node-2] loaded module [lang-painless]
[2019-04-26T13:45:20,742][INFO ][o.e.p.PluginsService ] [node-2] loaded module [parent-join]
[2019-04-26T13:45:20,742][INFO ][o.e.p.PluginsService ] [node-2] loaded module [percolator]
[2019-04-26T13:45:20,743][INFO ][o.e.p.PluginsService ] [node-2] loaded module [reindex]
[2019-04-26T13:45:20,743][INFO ][o.e.p.PluginsService ] [node-2] loaded module [transport-netty3]
[2019-04-26T13:45:20,743][INFO ][o.e.p.PluginsService ] [node-2] loaded module [transport-netty4]
[2019-04-26T13:45:20,743][INFO ][o.e.p.PluginsService ] [node-2] no plugins loaded
[2019-04-26T13:45:23,548][INFO ][o.e.d.DiscoveryModule ] [node-2] using discovery type [zen]
[2019-04-26T13:45:24,625][INFO ][o.e.n.Node ] [node-2] initialized
[2019-04-26T13:45:24,625][INFO ][o.e.n.Node ] [node-2] starting ...
[2019-04-26T13:45:24,881][INFO ][o.e.t.TransportService ] [node-2] publish_address {192.168.1.10:9302}, bound_addresses {192.168.1.10:9302}
[2019-04-26T13:45:24,901][INFO ][o.e.b.BootstrapChecks ] [node-2] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-04-26T13:45:31,310][INFO ][o.e.d.z.ZenDiscovery ] [node-2] failed to send join request to master [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}], reason [RemoteTransportException[[node-1][192.168.1.10:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}] not master for join request]; ], tried [3] times
[2019-04-26T13:45:34,528][INFO ][o.e.d.z.ZenDiscovery ] [node-2] failed to send join request to master [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}], reason [RemoteTransportException[[node-1][192.168.1.10:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}] not master for join request]; ], tried [3] times
[2019-04-26T13:45:40,745][INFO ][o.e.d.z.ZenDiscovery ] [node-2] failed to send join request to master [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}], reason [RemoteTransportException[[node-1][192.168.1.10:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}] not master for join request]; ], tried [3] times
[2019-04-26T13:45:46,965][INFO ][o.e.d.z.ZenDiscovery ] [node-2] failed to send join request to master [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}], reason [RemoteTransportException[[node-1][192.168.1.10:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}] not master for join request]; ], tried [3] times
[2019-04-26T13:45:53,195][INFO ][o.e.d.z.ZenDiscovery ] [node-2] failed to send join request to master [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}], reason [RemoteTransportException[[node-1][192.168.1.10:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{node-1}{b-BnNdqlTc2aoExHinDR9w}{vTRQJ-4oQlyDNraVddVL_w}{192.168.1.10}{192.168.1.10:9300}] not master for join request]; ], tried [3] times
[2019-04-26T13:45:55,022][WARN ][o.e.n.Node ] [node-2] timed out while waiting for initial discovery state - timeout: 30s
[2019-04-26T13:45:55,029][INFO ][o.e.h.n.Netty4HttpServerTransport] [node-2] publish_address {192.168.1.10:9202}, bound_addresses {192.168.1.10:9202}
[2019-04-26T13:45:55,029][INFO ][o.e.n.Node ] [node-2] started

9

These nodes have the same node ID. I think this means you made a copy of the data folder. You shouldn't do this. Start one of the nodes with an empty data folder instead.

10

Thanks dear David, your diag is correct !

So, to summary, the minimum cluster config on the same machine (for testing of course !!!) :

node1 : elasticsearch.yml
cluster.name: cluster-es
node.name: node-1
network.host: 192.168.1.10
http.port: 9200
transport.tcp.port: 9300
discovery.zen.ping.unicast.hosts: ["192.168.1.10:9300","192.168.1.10:9302"]
discovery.zen.minimum_master_nodes: 2

node2 : elasticsearch.yml
cluster.name: cluster-es
node.name: node-2
network.host: 192.168.1.10
http.port: 9202
transport.tcp.port: 9302
discovery.zen.ping.unicast.hosts: ["192.168.1.10:9300","192.168.1.10:9302"]
discovery.zen.minimum_master_nodes: 2

AND IMPORTANT thing is to empty the data in one of the node : /var/lib/elasticsearch/nodes

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.