How to set up a cluster locally with multiple VM's?

Elastic Stack Elasticsearch
1

Note: I'm just doing this to play around with and understand clustering while in development.

I have two instances of ES 5.6 running on two separate Ubuntu 16.04 virtual machines (that are on the same Windows host).

I'm trying to get the two instances of ES to be a part of the same cluster and for replica shards of one to appear on the other. In each Ubuntu machine, I modified the elasticsearch.yml so that discovery.zen.ping.unicast.hosts has the IP address of the other machine. (They can each curl the other by IP address).

I gave them the same cluster name and different node names. When each instance comes up, they show the same cluster name, as expected, but the cluster UUID is different for each, leading me to believe they don't know the other instance exists.

How can I get these two separate instances of ES, running on separate servers to join the same cluster?

1 Like
2

Can you show the configs from both nodes as well as the logs from when you start the node?

Please make sure you format the info using the code button - </> :slight_smile:

3

Config from the first node:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: search-cluster-dev
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: dev-node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 0.0.0.0
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["192.168.33.93"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

Config from the second node is very similar, with only the following differences:

node.name: dev-node-2
discovery.zen.ping.unicast.hosts: ["192.168.33.92"]

I'll post the log messages separately.

4

Log messages from dev-node-1

elasticsearch.log:

[2017-11-03T14:59:15,967][INFO ][o.e.n.Node               ] [] initializing ...
[2017-11-03T14:59:16,121][INFO ][o.e.e.NodeEnvironment    ] [-8-B5bw] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.5gb], net total_space [39.3gb], spins? [possibly], types [ext4]
[2017-11-03T14:59:16,122][INFO ][o.e.e.NodeEnvironment    ] [-8-B5bw] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-11-03T14:59:16,127][INFO ][o.e.n.Node               ] node name [-8-B5bw] derived from node ID [-8-B5bwuSvuFBABOLL6fcw]; set [node.name] to override
[2017-11-03T14:59:16,127][INFO ][o.e.n.Node               ] version[5.6.3], pid[12644], build[1a2f265/2017-10-06T20:33:39.012Z], OS[Linux/3.13.0-133-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2017-11-03T14:59:16,128][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-11-03T14:59:17,696][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [aggs-matrix-stats]
[2017-11-03T14:59:17,696][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [ingest-common]
[2017-11-03T14:59:17,696][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [lang-expression]
[2017-11-03T14:59:17,696][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [lang-groovy]
[2017-11-03T14:59:17,696][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [lang-mustache]
[2017-11-03T14:59:17,696][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [lang-painless]
[2017-11-03T14:59:17,697][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [parent-join]
[2017-11-03T14:59:17,697][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [percolator]
[2017-11-03T14:59:17,703][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [reindex]
[2017-11-03T14:59:17,703][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [transport-netty3]
[2017-11-03T14:59:17,703][INFO ][o.e.p.PluginsService     ] [-8-B5bw] loaded module [transport-netty4]
[2017-11-03T14:59:17,704][INFO ][o.e.p.PluginsService     ] [-8-B5bw] no plugins loaded
[2017-11-03T14:59:20,648][INFO ][o.e.d.DiscoveryModule    ] [-8-B5bw] using discovery type [zen]
[2017-11-03T14:59:21,617][INFO ][o.e.n.Node               ] initialized
[2017-11-03T14:59:21,617][INFO ][o.e.n.Node               ] [-8-B5bw] starting ...
[2017-11-03T14:59:21,851][INFO ][o.e.t.TransportService   ] [-8-B5bw] publish_address {10.0.2.15:9300}, bound_addresses {[::]:9300}
[2017-11-03T14:59:21,867][INFO ][o.e.b.BootstrapChecks    ] [-8-B5bw] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-11-03T14:59:24,966][INFO ][o.e.c.s.ClusterService   ] [-8-B5bw] new_master {-8-B5bw}{-8-B5bwuSvuFBABOLL6fcw}{7du1pYw_QlGF4aZS7i2tDg}{10.0.2.15}{10.0.2.15:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-11-03T14:59:25,019][INFO ][o.e.h.n.Netty4HttpServerTransport] [-8-B5bw] publish_address {10.0.2.15:9200}, bound_addresses {[::]:9200}
[2017-11-03T14:59:25,020][INFO ][o.e.n.Node               ] [-8-B5bw] started
[2017-11-03T14:59:25,021][INFO ][o.e.g.GatewayService     ] [-8-B5bw] recovered [0] indices into cluster_state
[2017-11-03T15:21:24,196][INFO ][o.e.n.Node               ] [-8-B5bw] stopping ...
[2017-11-03T15:21:24,210][INFO ][o.e.n.Node               ] [-8-B5bw] stopped
[2017-11-03T15:21:24,210][INFO ][o.e.n.Node               ] [-8-B5bw] closing ...
[2017-11-03T15:21:24,225][INFO ][o.e.n.Node               ] [-8-B5bw] closed
5

From node-dev-1
(Sorry for the multiple posts, the character limit won't let me post all at once)

search-cluster-dev.log:

[2017-11-05T16:22:33,991][INFO ][o.e.n.Node               ] [dev-node-1] initializing ...
[2017-11-05T16:22:34,207][INFO ][o.e.e.NodeEnvironment    ] [dev-node-1] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.5gb], net total_space [39.3gb], spins? [possibly], types [ext4]
[2017-11-05T16:22:34,208][INFO ][o.e.e.NodeEnvironment    ] [dev-node-1] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-11-05T16:22:34,250][INFO ][o.e.n.Node               ] [dev-node-1] node name [dev-node-1], node ID [-8-B5bwuSvuFBABOLL6fcw]
[2017-11-05T16:22:34,251][INFO ][o.e.n.Node               ] [dev-node-1] version[5.6.3], pid[2115], build[1a2f265/2017-10-06T20:33:39.012Z], OS[Linux/3.13.0-133-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2017-11-05T16:22:34,251][INFO ][o.e.n.Node               ] [dev-node-1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-11-05T16:22:36,478][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [aggs-matrix-stats]
[2017-11-05T16:22:36,478][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [ingest-common]
[2017-11-05T16:22:36,479][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [lang-expression]
[2017-11-05T16:22:36,479][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [lang-groovy]
[2017-11-05T16:22:36,479][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [lang-mustache]
[2017-11-05T16:22:36,479][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [lang-painless]
[2017-11-05T16:22:36,479][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [parent-join]
[2017-11-05T16:22:36,479][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [percolator]
[2017-11-05T16:22:36,479][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [reindex]
[2017-11-05T16:22:36,479][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [transport-netty3]
[2017-11-05T16:22:36,483][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [transport-netty4]
[2017-11-05T16:22:36,483][INFO ][o.e.p.PluginsService     ] [dev-node-1] no plugins loaded
[2017-11-05T16:22:41,305][INFO ][o.e.d.DiscoveryModule    ] [dev-node-1] using discovery type [zen]
[2017-11-05T16:22:43,144][INFO ][o.e.n.Node               ] [dev-node-1] initialized
[2017-11-05T16:22:43,144][INFO ][o.e.n.Node               ] [dev-node-1] starting ...
[2017-11-05T16:22:43,610][INFO ][o.e.t.TransportService   ] [dev-node-1] publish_address {10.0.2.15:9300}, bound_addresses {[::]:9300}
[2017-11-05T16:22:43,648][INFO ][o.e.b.BootstrapChecks    ] [dev-node-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-11-05T16:22:46,786][INFO ][o.e.c.s.ClusterService   ] [dev-node-1] new_master {dev-node-1}{-8-B5bwuSvuFBABOLL6fcw}{6FvyruzUTI6ZcJggCM9uPQ}{10.0.2.15}{10.0.2.15:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-11-05T16:22:46,861][INFO ][o.e.h.n.Netty4HttpServerTransport] [dev-node-1] publish_address {10.0.2.15:9200}, bound_addresses {[::]:9200}
[2017-11-05T16:22:46,861][INFO ][o.e.n.Node               ] [dev-node-1] started
[2017-11-05T16:22:47,239][INFO ][o.e.g.GatewayService     ] [dev-node-1] recovered [1] indices into cluster_state
[2017-11-05T16:22:47,813][INFO ][o.e.c.r.a.AllocationService] [dev-node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[marvel][4]] ...]).
6

From node-dev-2

search-cluster-dev.log

[2017-11-05T16:30:03,672][INFO ][o.e.n.Node               ] [dev-node-2] initializing ...
[2017-11-05T16:30:03,868][INFO ][o.e.e.NodeEnvironment    ] [dev-node-2] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.5gb], net total_space [39.3gb], spins? [possibly], types [ext4]
[2017-11-05T16:30:03,871][INFO ][o.e.e.NodeEnvironment    ] [dev-node-2] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-11-05T16:30:03,872][INFO ][o.e.n.Node               ] [dev-node-2] node name [dev-node-2], node ID [BesMEgDtSEi7uifonPftTg]
[2017-11-05T16:30:03,874][INFO ][o.e.n.Node               ] [dev-node-2] version[5.6.3], pid[2122], build[1a2f265/2017-10-06T20:33:39.012Z], OS[Linux/3.13.0-133-generic/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2017-11-05T16:30:03,875][INFO ][o.e.n.Node               ] [dev-node-2] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [aggs-matrix-stats]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [ingest-common]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [lang-expression]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [lang-groovy]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [lang-mustache]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [lang-painless]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [parent-join]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [percolator]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [reindex]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [transport-netty3]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [transport-netty4]
[2017-11-05T16:30:05,373][INFO ][o.e.p.PluginsService     ] [dev-node-2] no plugins loaded
[2017-11-05T16:30:07,872][INFO ][o.e.d.DiscoveryModule    ] [dev-node-2] using discovery type [zen]
[2017-11-05T16:30:08,804][INFO ][o.e.n.Node               ] [dev-node-2] initialized
[2017-11-05T16:30:08,804][INFO ][o.e.n.Node               ] [dev-node-2] starting ...
[2017-11-05T16:30:09,044][INFO ][o.e.t.TransportService   ] [dev-node-2] publish_address {10.0.2.15:9300}, bound_addresses {[::]:9300}
[2017-11-05T16:30:09,061][INFO ][o.e.b.BootstrapChecks    ] [dev-node-2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-11-05T16:30:12,118][INFO ][o.e.c.s.ClusterService   ] [dev-node-2] new_master {dev-node-2}{BesMEgDtSEi7uifonPftTg}{sVAeq0OUQbqYT_xI8IP_ng}{10.0.2.15}{10.0.2.15:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-11-05T16:30:12,188][INFO ][o.e.g.GatewayService     ] [dev-node-2] recovered [0] indices into cluster_state
[2017-11-05T16:30:12,191][INFO ][o.e.h.n.Netty4HttpServerTransport] [dev-node-2] publish_address {10.0.2.15:9200}, bound_addresses {[::]:9200}
[2017-11-05T16:30:12,191][INFO ][o.e.n.Node               ] [dev-node-2] started
7

The publish address is 10.0.2.15 not 192.168.33.92.
May be change that?

8

@dadoonet
Why 10.0.2.15? I thought it was supposed to be a list IP's of other hosts running elasticsearch on the same network?

9

Could you try it please and share the logs once you're done?

10

Changed both discovery.zen.ping.unicast.hosts fields in both machines to 10.0.2.15.

From machine node-dev-1, the only log that was updated was search-cluster-dev.log, with the following lines:

[2017-11-05T18:29:52,287][INFO ][o.e.n.Node               ] [dev-node-1] stopping ...
[2017-11-05T18:29:52,327][INFO ][o.e.n.Node               ] [dev-node-1] stopped
[2017-11-05T18:29:52,327][INFO ][o.e.n.Node               ] [dev-node-1] closing ...
[2017-11-05T18:29:52,348][INFO ][o.e.n.Node               ] [dev-node-1] closed
[2017-11-05T18:30:51,818][INFO ][o.e.n.Node               ] [dev-node-1] initializing ...
[2017-11-05T18:30:51,971][INFO ][o.e.e.NodeEnvironment    ] [dev-node-1] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.5gb], net total_space [39.3gb], spins? [possibl
y], types [ext4]
[2017-11-05T18:30:51,971][INFO ][o.e.e.NodeEnvironment    ] [dev-node-1] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-11-05T18:30:51,983][INFO ][o.e.n.Node               ] [dev-node-1] node name [dev-node-1], node ID [-8-B5bwuSvuFBABOLL6fcw]
[2017-11-05T18:30:51,984][INFO ][o.e.n.Node               ] [dev-node-1] version[5.6.3], pid[2404], build[1a2f265/2017-10-06T20:33:39.012Z], OS[Linux/3.13.0-133-generic/amd64], JVM[Oracle C
orporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2017-11-05T18:30:51,984][INFO ][o.e.n.Node               ] [dev-node-1] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiating
OccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.nett
y.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryErro
r, -Des.path.home=/usr/share/elasticsearch]
[2017-11-05T18:30:53,465][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [aggs-matrix-stats]
[2017-11-05T18:30:53,465][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [ingest-common]
[2017-11-05T18:30:53,465][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [lang-expression]
[2017-11-05T18:30:53,465][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [lang-groovy]
[2017-11-05T18:30:53,465][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [lang-mustache]
[2017-11-05T18:30:53,465][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [lang-painless]
[2017-11-05T18:30:53,470][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [parent-join]
[2017-11-05T18:30:53,470][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [percolator]
[2017-11-05T18:30:53,471][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [reindex]
[2017-11-05T18:30:53,471][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [transport-netty3]
[2017-11-05T18:30:53,471][INFO ][o.e.p.PluginsService     ] [dev-node-1] loaded module [transport-netty4]
[2017-11-05T18:30:53,471][INFO ][o.e.p.PluginsService     ] [dev-node-1] no plugins loaded
[2017-11-05T18:30:56,176][INFO ][o.e.d.DiscoveryModule    ] [dev-node-1] using discovery type [zen]
[2017-11-05T18:30:57,296][INFO ][o.e.n.Node               ] [dev-node-1] initialized
[2017-11-05T18:30:57,296][INFO ][o.e.n.Node               ] [dev-node-1] starting ...
[2017-11-05T18:30:57,561][INFO ][o.e.t.TransportService   ] [dev-node-1] publish_address {10.0.2.15:9300}, bound_addresses {[::]:9300}
[2017-11-05T18:30:57,588][INFO ][o.e.b.BootstrapChecks    ] [dev-node-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-11-05T18:31:00,754][INFO ][o.e.c.s.ClusterService   ] [dev-node-1] new_master {dev-node-1}{-8-B5bwuSvuFBABOLL6fcw}{lYHroCd_QrWKR-BtPtn6dQ}{10.0.2.15}{10.0.2.15:9300}, reason: zen-disc
o-elected-as-master ([0] nodes joined)
[2017-11-05T18:31:00,874][INFO ][o.e.h.n.Netty4HttpServerTransport] [dev-node-1] publish_address {10.0.2.15:9200}, bound_addresses {[::]:9200}
[2017-11-05T18:31:00,874][INFO ][o.e.n.Node               ] [dev-node-1] started
[2017-11-05T18:31:01,236][INFO ][o.e.g.GatewayService     ] [dev-node-1] recovered [1] indices into cluster_state
[2017-11-05T18:31:01,741][INFO ][o.e.c.r.a.AllocationService] [dev-node-1] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[marvel][4]] ...]).
11

Similarly after changing to 10.0.2.15, the search-cluster-dev.log file from node-dev-2:

[2017-11-05T18:32:37,862][INFO ][o.e.n.Node               ] [dev-node-2] stopping ...
[2017-11-05T18:32:37,883][INFO ][o.e.n.Node               ] [dev-node-2] stopped
[2017-11-05T18:32:37,884][INFO ][o.e.n.Node               ] [dev-node-2] closing ...
[2017-11-05T18:32:37,905][INFO ][o.e.n.Node               ] [dev-node-2] closed
[2017-11-05T18:32:39,775][INFO ][o.e.n.Node               ] [dev-node-2] initializing ...
[2017-11-05T18:32:39,925][INFO ][o.e.e.NodeEnvironment    ] [dev-node-2] using [1] data paths, mounts [[/ (/dev/sda1)]], net usable_space [35.5gb], net total_space [39.3gb], spins? [possibl
y], types [ext4]
[2017-11-05T18:32:39,925][INFO ][o.e.e.NodeEnvironment    ] [dev-node-2] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-11-05T18:32:39,927][INFO ][o.e.n.Node               ] [dev-node-2] node name [dev-node-2], node ID [BesMEgDtSEi7uifonPftTg]
[2017-11-05T18:32:39,927][INFO ][o.e.n.Node               ] [dev-node-2] version[5.6.3], pid[2281], build[1a2f265/2017-10-06T20:33:39.012Z], OS[Linux/3.13.0-133-generic/amd64], JVM[Oracle C
orporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_151/25.151-b12]
[2017-11-05T18:32:39,928][INFO ][o.e.n.Node               ] [dev-node-2] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiating
OccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.nett
y.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryErro
r, -Des.path.home=/usr/share/elasticsearch]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [aggs-matrix-stats]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [ingest-common]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [lang-expression]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [lang-groovy]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [lang-mustache]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [lang-painless]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [parent-join]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [percolator]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [reindex]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [transport-netty3]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] loaded module [transport-netty4]
[2017-11-05T18:32:41,414][INFO ][o.e.p.PluginsService     ] [dev-node-2] no plugins loaded
[2017-11-05T18:32:44,623][INFO ][o.e.d.DiscoveryModule    ] [dev-node-2] using discovery type [zen]
[2017-11-05T18:32:45,727][INFO ][o.e.n.Node               ] [dev-node-2] initialized
[2017-11-05T18:32:45,727][INFO ][o.e.n.Node               ] [dev-node-2] starting ...
[2017-11-05T18:32:46,005][INFO ][o.e.t.TransportService   ] [dev-node-2] publish_address {10.0.2.15:9300}, bound_addresses {[::]:9300}
[2017-11-05T18:32:46,027][INFO ][o.e.b.BootstrapChecks    ] [dev-node-2] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks
[2017-11-05T18:32:49,146][INFO ][o.e.c.s.ClusterService   ] [dev-node-2] new_master {dev-node-2}{BesMEgDtSEi7uifonPftTg}{0VwPQnGjQ6mfG53qRLdn_Q}{10.0.2.15}{10.0.2.15:9300}, reason: zen-disc
o-elected-as-master ([0] nodes joined)
[2017-11-05T18:32:49,200][INFO ][o.e.g.GatewayService     ] [dev-node-2] recovered [0] indices into cluster_state
[2017-11-05T18:32:49,202][INFO ][o.e.h.n.Netty4HttpServerTransport] [dev-node-2] publish_address {10.0.2.15:9200}, bound_addresses {[::]:9200}
[2017-11-05T18:32:49,202][INFO ][o.e.n.Node               ] [dev-node-2] started

The nodes are currently showing different UUID's for their clusters.

12

Any firewall in the middle? May be pings are blocked.

Can you change the log4j property file and add:

logger.discovery.name = org.elasticsearch.discovery
logger.discovery.level = trace

And restart and paste the logs?

13

No firewall between that I'm aware of. They are two VM's on the same Windows host and (right now) I'm on my home network, where I've never made any special firewall configurations.

I see a file called log4j2.properties, and added the lines specified above (these properties were not already defined).

New log output on search-cluster-dev.log on Node node-dev-1 is in this paste bin. (It was more than 7000 characters so I can't post it directly here).

https://pastebin.com/AtbVPUdN

Same log on node-dev-2: https://pastebin.com/mvwMyZPt

14

I see.

Can you set again to

discovery.zen.ping.unicast.hosts: ["192.168.33.92", "192.168.33.93"]

And change for server1:

network.host: 192.168.33.92

And change for server2:

network.host: 192.168.33.93

And restart?

15

I think that worked! :slight_smile:
Node 1's log initially had some exceptions but after I brought up Node 2, I noticed they appear to be connected to the same cluster now (by UUID).

Node 1's log: https://pastebin.com/g9CZ9mTv

Node 2's log: https://pastebin.com/KzCgHGXQ

Two node cluster
Two node cluster.PNG731×260 17.9 KB

Thank you!

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.