Windows ELK 7.1 2-node cluster

[zen.xen]

Hello,
I try to create 2-node cluster with ELK 7.1 on Windows in my lab but I have troubles and it doesn't work.
Configuration of node-01 is:

cluster.name: elasticsearch

node.name: node-01
node.master: true
node.data: true

network.host: 0.0.0.0
network.publish_host: 10.0.0.1

http.port: 9200
transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2"]
discovery.zen.minimum_master_nodes: 2

Configuration of node-02 is:

cluster.name: elasticsearch

node.name: node-02
node.master: true
node.data: true

network.host: 0.0.0.0
network.publish_host: 10.0.0.2

http.port: 9200
transport.tcp.port: 9300

discovery.zen.ping.unicast.hosts: ["10.0.0.1", "10.0.0.2"]
discovery.zen.minimum_master_nodes: 2

What is wrong with this configuration? Status is yellow.

[dadoonet (David Pilato)]

Firewall? The IP address elasticsearch is not the one you defined?

Logs would tell more I think.

[zen.xen]

Firewall is off on both machines. What do you mean

The IP address elasticsearch is not the one you defined?

Both nodes has only one IP address.

node-01 elasticsearch.log

[2019-05-31T12:02:50,437][INFO ][o.e.e.NodeEnvironment    ] [node-01] using [1] data paths, mounts [[(C:)]], net usable_space [31.9gb], net total_space [59.6gb], types [NTFS]
[2019-05-31T12:02:50,437][INFO ][o.e.e.NodeEnvironment    ] [node-01] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-05-31T12:02:50,469][INFO ][o.e.n.Node               ] [node-01] node name [node-01], node ID [zX2VwQ6cS3GfKNiKDuRhSg], cluster name [elasticsearch]
[2019-05-31T12:02:50,469][INFO ][o.e.n.Node               ] [node-01] version[7.1.0], pid[3476], build[oss/zip/606a173/2019-05-16T00:43:15.323135Z], OS[Windows Server 2012/6.2/amd64], JVM[/OpenJDK 64-Bit Server VM/1.8.0_212-1-ojdkbuild/25.212-b04]
[2019-05-31T12:02:50,469][INFO ][o.e.n.Node               ] [node-01] JVM home [C:\Bitnami\elk-7.1.0-0\java]
[2019-05-31T12:02:50,469][INFO ][o.e.n.Node               ] [node-01] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss320k, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Dio.netty.allocator.type=unpooled, -Delasticsearch, -Des.path.home=C:\Bitnami\elk-7.1.0-0\elasticsearch, -Des.path.conf=C:\Bitnami\elk-7.1.0-0\elasticsearch\config, -Des.distribution.flavor=oss, -Des.distribution.type=zip, -Des.bundled_jdk=true, exit, abort, -Xms1024m, -Xmx1024m, -Xss320k]
[2019-05-31T12:02:51,252][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [aggs-matrix-stats]
[2019-05-31T12:02:51,253][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [analysis-common]
[2019-05-31T12:02:51,253][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [ingest-common]
[2019-05-31T12:02:51,253][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [ingest-geoip]
[2019-05-31T12:02:51,253][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [ingest-user-agent]
[2019-05-31T12:02:51,253][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [lang-expression]
[2019-05-31T12:02:51,253][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [lang-mustache]
[2019-05-31T12:02:51,255][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [lang-painless]
[2019-05-31T12:02:51,255][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [mapper-extras]
[2019-05-31T12:02:51,255][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [parent-join]
[2019-05-31T12:02:51,255][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [percolator]
[2019-05-31T12:02:51,255][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [rank-eval]
[2019-05-31T12:02:51,255][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [reindex]
[2019-05-31T12:02:51,255][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [repository-url]
[2019-05-31T12:02:51,255][INFO ][o.e.p.PluginsService     ] [node-01] loaded module [transport-netty4]
[2019-05-31T12:02:51,255][INFO ][o.e.p.PluginsService     ] [node-01] no plugins loaded
[2019-05-31T12:02:54,863][INFO ][o.e.d.DiscoveryModule    ] [node-01] using discovery type [zen] and seed hosts providers [settings]
[2019-05-31T12:02:55,317][INFO ][o.e.n.Node               ] [node-01] initialized
[2019-05-31T12:02:55,317][INFO ][o.e.n.Node               ] [node-01] starting ...
[2019-05-31T12:02:56,018][INFO ][o.e.t.TransportService   ] [node-01] publish_address {10.0.0.1:9300}, bound_addresses {[::]:9300}
[2019-05-31T12:02:56,034][INFO ][o.e.b.BootstrapChecks    ] [node-01] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-05-31T12:02:56,034][INFO ][o.e.c.c.Coordinator      ] [node-01] cluster UUID [cbTvi_vPSNCPCIMVdPU5_A]
[2019-05-31T12:02:56,221][INFO ][o.e.c.s.MasterService    ] [node-01] elected-as-master ([1] nodes joined)[{node-01}{zX2VwQ6cS3GfKNiKDuRhSg}{Kj0VxpSGTWqUUyczPUiqXQ}{10.0.0.1}{10.0.0.1:9300} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 17, version: 76, reason: master node changed {previous [], current [{node-01}{zX2VwQ6cS3GfKNiKDuRhSg}{Kj0VxpSGTWqUUyczPUiqXQ}{10.0.0.1}{10.0.0.1:9300}]}
[2019-05-31T12:02:56,271][INFO ][o.e.c.s.ClusterApplierService] [node-01] master node changed {previous [], current [{node-01}{zX2VwQ6cS3GfKNiKDuRhSg}{Kj0VxpSGTWqUUyczPUiqXQ}{10.0.0.1}{10.0.0.1:9300}]}, term: 17, version: 76, reason: Publication{term=17, version=76}
[2019-05-31T12:02:56,380][INFO ][o.e.g.GatewayService     ] [node-01] recovered [2] indices into cluster_state
[2019-05-31T12:02:56,412][INFO ][o.e.h.AbstractHttpServerTransport] [node-01] publish_address {10.0.0.1:9200}, bound_addresses {[::]:9200}
[2019-05-31T12:02:56,412][INFO ][o.e.n.Node               ] [node-01] started
[2019-05-31T12:02:57,051][INFO ][o.e.c.r.a.AllocationService] [node-01] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana_1][0]] ...]).

[zen.xen]

node-02 elasticsearch.log

[2019-05-31T12:02:57,624][INFO ][o.e.e.NodeEnvironment    ] [node-02] heap size [989.8mb], compressed ordinary object pointers [true]
[2019-05-31T12:02:57,656][INFO ][o.e.n.Node               ] [node-02] node name [node-02], node ID [0p-cIW3qT6yeFvGNE7kCKw], cluster name [elasticsearch]
[2019-05-31T12:02:57,656][INFO ][o.e.n.Node               ] [node-02] version[7.1.0], pid[1436], build[oss/zip/606a173/2019-05-16T00:43:15.323135Z], OS[Windows Server 2012/6.2/amd64], JVM[/OpenJDK 64-Bit Server VM/1.8.0_212-1-ojdkbuild/25.212-b04]
[2019-05-31T12:02:57,656][INFO ][o.e.n.Node               ] [node-02] JVM home [C:\Bitnami\elk-7.1.0-0\java]
[2019-05-31T12:02:57,656][INFO ][o.e.n.Node               ] [node-02] JVM arguments [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss320k, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.io.tmpdir=C:\Users\ADMINI~1\AppData\Local\Temp\elasticsearch, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -XX:+PrintGCDetails, -XX:+PrintGCDateStamps, -XX:+PrintTenuringDistribution, -XX:+PrintGCApplicationStoppedTime, -Xloggc:logs/gc.log, -XX:+UseGCLogFileRotation, -XX:NumberOfGCLogFiles=32, -XX:GCLogFileSize=64m, -Dio.netty.allocator.type=unpooled, -Delasticsearch, -Des.path.home=C:\Bitnami\elk-7.1.0-0\elasticsearch, -Des.path.conf=C:\Bitnami\elk-7.1.0-0\elasticsearch\config, -Des.distribution.flavor=oss, -Des.distribution.type=zip, -Des.bundled_jdk=true, exit, abort, -Xms1024m, -Xmx1024m, -Xss320k]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [aggs-matrix-stats]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [analysis-common]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [ingest-common]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [ingest-geoip]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [ingest-user-agent]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [lang-expression]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [lang-mustache]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [lang-painless]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [mapper-extras]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [parent-join]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [percolator]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [rank-eval]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [reindex]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [repository-url]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] loaded module [transport-netty4]
[2019-05-31T12:02:58,796][INFO ][o.e.p.PluginsService     ] [node-02] no plugins loaded
[2019-05-31T12:03:03,485][INFO ][o.e.d.DiscoveryModule    ] [node-02] using discovery type [zen] and seed hosts providers [settings]
[2019-05-31T12:03:04,031][INFO ][o.e.n.Node               ] [node-02] initialized
[2019-05-31T12:03:04,031][INFO ][o.e.n.Node               ] [node-02] starting ...
[2019-05-31T12:03:04,625][INFO ][o.e.t.TransportService   ] [node-02] publish_address {10.0.0.2:9300}, bound_addresses {[::]:9300}
[2019-05-31T12:03:04,641][INFO ][o.e.b.BootstrapChecks    ] [node-02] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-05-31T12:03:04,641][INFO ][o.e.c.c.Coordinator      ] [node-02] cluster UUID [Cw4mtXLBQui1K1NlIDCyUg]
[2019-05-31T12:03:04,799][INFO ][o.e.c.s.MasterService    ] [node-02] elected-as-master ([1] nodes joined)[{node-02}{0p-cIW3qT6yeFvGNE7kCKw}{Xhr5RBRwSFWb6k9ycGtYPQ}{10.0.0.2}{10.0.0.2:9300} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 16, version: 44, reason: master node changed {previous [], current [{node-02}{0p-cIW3qT6yeFvGNE7kCKw}{Xhr5RBRwSFWb6k9ycGtYPQ}{10.0.0.2}{10.0.0.2:9300}]}
[2019-05-31T12:03:04,845][INFO ][o.e.c.s.ClusterApplierService] [node-02] master node changed {previous [], current [{node-02}{0p-cIW3qT6yeFvGNE7kCKw}{Xhr5RBRwSFWb6k9ycGtYPQ}{10.0.0.2}{10.0.0.2:9300}]}, term: 16, version: 44, reason: Publication{term=16, version=44}
[2019-05-31T12:03:04,861][INFO ][o.e.h.AbstractHttpServerTransport] [node-02] publish_address {10.0.0.2:9200}, bound_addresses {[::]:9200}
[2019-05-31T12:03:04,861][INFO ][o.e.n.Node               ] [node-02] started
[2019-05-31T12:03:04,908][INFO ][o.e.g.GatewayService     ] [node-02] recovered [2] indices into cluster_state
[2019-05-31T12:03:05,442][INFO ][o.e.c.r.a.AllocationService] [node-02] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[.kibana_1][0]] ...]).

[zen.xen]

I have discovered what was wrong with my cluster. I have analyzed logs on both machines and found something like this:

join validation on cluster state with a different cluster uuid LcitTe7KQFCEF8Ww_MuU_g than local cluster uuid or6tpJqPRqSzOdea_eYA7g, rejecting

According this link I have checked data path ..\elasticsearch\data\nodes\0 there were some data, I deleted them all, restarted ELK stack and cluster started working and changed status from yellow to green.