Cluster Setup 3 Node Cluster problem

[thev0yager]

This is from the running node that is my master

[2019-07-03T19:13:35,835][INFO ][o.e.n.Node               ] [corp-elk02] initialized
[2019-07-03T19:13:35,835][INFO ][o.e.n.Node               ] [corp-elk02] starting ...
[2019-07-03T19:13:36,075][INFO ][o.e.t.TransportService   ] [corp-elk02] publish_address {10.101.0.141:9300}, bound_addresses {10.101.0.141:9300}
[2019-07-03T19:13:36,104][INFO ][o.e.b.BootstrapChecks    ] [corp-elk02] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-07-03T19:13:36,125][INFO ][o.e.c.c.Coordinator      ] [corp-elk02] cluster UUID [u4y2xk7mSv2x2_a8l9k2ew]
[2019-07-03T19:13:36,337][INFO ][o.e.c.s.MasterService    ] [corp-elk02] elected-as-master ([1] nodes joined)[{corp-elk02}{PpIwvyyXREKSn0HscUCmag}{xvLmIlfWRvWd4NnFPnqzPA}{10.101.0.141}{10.101.0.141:9300}{ml.machine_memory=16820158464, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 13, version: 66, reason: master node changed {previous [], current [{corp-elk02}{PpIwvyyXREKSn0HscUCmag}{xvLmIlfWRvWd4NnFPnqzPA}{10.101.0.141}{10.101.0.141:9300}{ml.machine_memory=16820158464, xpack.installed=true, ml.max_open_jobs=20}]}

[thev0yager]

[2019-07-03T14:21:51,198][INFO ][o.e.n.Node               ] [choice-node1] initialized
[2019-07-03T14:21:51,201][INFO ][o.e.n.Node               ] [choice-node1] starting ...
[2019-07-03T14:21:51,379][INFO ][o.e.t.TransportService   ] [choice-node1] publish_address {10.101.0.140:9300}, bound_addresses {10.101.0.140:9300}
[2019-07-03T14:21:51,391][INFO ][o.e.b.BootstrapChecks    ] [choice-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2019-07-03T14:21:51,403][INFO ][o.e.c.c.Coordinator      ] [choice-node1] cluster UUID [TUPCVI9iQBK5p3Erp3L5ew]
[2019-07-03T14:22:01,417][WARN ][o.e.c.c.ClusterFormationFailureHelper] [choice-node1] master not discovered yet: have discovered []; discovery will continue using [10.101.0.140:9200, 10.101.0.141:9200] from hosts providers and [{choice-node1}{ukRS6dHuTiWJVjVvpSIKBQ}{IxzRiArITDaqHglg4Exb-Q}{10.101.0.140}{10.101.0.140:9300}{ml.machine_memory=16820174848, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 52, last-accepted version 96 in term 52
[2019-07-03T14:22:09,433][WARN ][o.e.t.TcpTransport       ] [choice-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.101.0.140:37908, remoteAddress=/10.101.0.141:9200}], closing connection
io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50)

This is the second node I am getting the same error again "master not discovered yet".

[DavidTurner]

Ok, the discovery addresses in the second node are still using port 9200, and the node name is a different format, so I think you're not using the right config file (or at least you're not sharing the right config file with us).

Also the cluster UUID doesn't match, indicating that this node belonged to a different cluster (possibly on its own) earlier in its life. You need to shut this node down, fix the config, wipe its data, and then it should join the cluster.

[userguy]

I think I agree with @DavidTurner . If it is new cluster . Can you start fresh .

1- Please keep config file same on all the host . Except hostname or node.name parameter and fall back to minimum config

Also check your selinux config and firewall setting/iptables setting .

[elasticforme (Sachin)]

@DavidTurner

Also the cluster UUID doesn't match, indicating that this node belonged to a different cluster (possibly on its own) earlier in its life. You need to shut this node down, fix the config, wipe its data, and then it should join the cluster.

Completly agree,
stop everything, delete all dir which has your elasticsearch data. and start fresh.
turn on daemon on all three node with correct config file and it should form cluster.

[thev0yager]

Okay after some struggles I was able to get the cluster working and another node joined up with it.

xpack.security.enabled: false

Deleting this line in my elasticsearch.yml file it just started working. And now I can run the cluster with a master and non-master. The thing I am troubled with now is how to get data from one point to another. I am getting an index coming through to kibana, but it doesn't give me any data. Right now I have a SQL server and I have heartbeats setup and running on that. I know I am getting data from it because when I look at logstash I am getting logs from heartbeats. I just don't know why the visualization isn't working.

[thev0yager]

Is what I am doing possible? I am 100% willing to do it the way your guys website is telling me to do I just don't want to keep on walking blind.

[DavidTurner]

I've no idea, sorry, this thread has wandered a long way from its original problem, which sounds like it's resolved. I would recommend starting a new thread and describing things again because it's almost impossible to work out what information here is still pertinent and what you have changed.