# Cluster Setup 3 Node Cluster problem **URL:** \ **Category:** Elasticsearch\ **Created:** [July 2, 2019, 2:04pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526 "2019-07-02T14:04:23Z")\ **Posts on this page:** 20\ **Page:** 2
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 5:51pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/21 "2019-07-03T17:51:18Z")
Okay I definitely need to set the story straight I am really sorry about all the confusion. I am trying to get to the point where I have a three node cluster. As of now I have had trouble with getting that 3rd node setup. A lot of unrelated issues at the moment. Also between the time I submitted this I did what you said and changed the port number to 9300. I will be submitting the logs that I am now getting from Elasticsearch in a second. Once I get a second node joined up in the is cluster I'll try and add the third node. Btw thank you so much for your help and sorry for the trouble. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 5:55pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/22 "2019-07-03T17:55:02Z")
``` [2019-07-03T12:51:47,945][INFO][o.e.n.Node] [choice-node1] initialized [2019-07-03T12:51:47,949][INFO][o.e.n.Node] [choice-node1] starting ... [2019-07-03T12:51:48,155][INFO][o.e.t.TransportService] [choice-node1] publish_address {10.101.0.140:9300}, bound_addresses {10.101.0.140:9300} [2019-07-03T12:51:48,167][INFO][o.e.b.BootstrapChecks] [choice-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-07-03T12:51:48,178][INFO][o.e.c.c.Coordinator] [choice-node1] cluster UUID [TUPCVI9iQBK5p3Erp3L5ew] [2019-07-03T12:51:58,215][WARN][o.e.c.c.ClusterFormationFailureHelper] [choice-node1] master not discovered yet: have discovered []; discovery will continue using [10.101.0.140:9200, 10.101.0.141:9200] from hosts providers and [{choice-node1}{ukRS6dHuTiWJVjVvpSIKBQ}{tQOZJeltRSiCHM7Osi0wnA}{10.101.0.140}{10.101.0.140:9300}{ml.machine_memory=16820174848, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 52, last-accepted version 96 in term 52 [2019-07-03T12:52:08,216][WARN][o.e.c.c.ClusterFormationFailureHelper] [choice-node1] master not discovered yet: have discovered []; discovery will continue using [10.101.0.140:9200, 10.101.0.141:9200] from hosts providers and [{choice-node1}{ukRS6dHuTiWJVjVvpSIKBQ}{tQOZJeltRSiCHM7Osi0wnA}{10.101.0.140}{10.101.0.140:9300}{ml.machine_memory=16820174848, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 52, last-accepted version 96 in term 52 [2019-07-03T12:52:18,219][WARN][o.e.c.c.ClusterFormationFailureHelper] [choice-node1] master not discovered yet: have discovered []; discovery will continue using [10.101.0.140:9200, 10.101.0.141:9200] from hosts providers and [{choice-node1}{ukRS6dHuTiWJVjVvpSIKBQ}{tQOZJeltRSiCHM7Osi0wnA}{10.101.0.140}{10.101.0.140:9300}{ml.machine_memory=16820174848, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 52, last-accepted version 96 in term 52 [2019-07-03T12:52:18,248][WARN][o.e.n.Node] [choice-node1] timed out while waiting for initial discovery state - timeout: 30s [2019-07-03T12:52:18,263][INFO][o.e.h.AbstractHttpServerTransport] [choice-node1] publish_address {10.101.0.140:9200}, bound_addresses {10.101.0.140:9200} [2019-07-03T12:52:18,264][INFO][o.e.n.Node] [choice-node1] started [2019-07-03T12:52:28,220][WARN][o.e.c.c.ClusterFormationFailureHelper] [choice-node1] master not discovered yet: have discovered []; discovery will continue using [10.101.0.140:9200, 10.101.0.141:9200] from hosts providers and [{choice-node1}{ukRS6dHuTiWJVjVvpSIKBQ}{tQOZJeltRSiCHM7Osi0wnA}{10.101.0.140}{10.101.0.140:9300}{ml.machine_memory=16820174848, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 52, last-accepted version 96 in term 52 [2019-07-03T12:52:37,248][WARN][o.e.t.TcpTransport] [choice-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.101.0.140:58134, remoteAddress=/10.101.0.140:9200}], closing connection io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50) at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-codec-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:241) [netty-handler-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:656) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:556) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:510) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:470) [netty-transport-4.1.32.Final.jar:4.1.32.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909) [netty-common-4.1.32.Final.jar:4.1.32.Final] at java.lang.Thread.run(Thread.java:835) [?:?] Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50) at org.elasticsearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:841) ~[elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.transport.TcpTransport.readMessageLength(TcpTransport.java:827) ~[elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:40) ~[transport-netty4-client-7.1.1.jar:7.1.1 ``` ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 5:57pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/23 "2019-07-03T17:57:04Z")
This is one of the nodes, and this is from the point where it starts up to a "caused by" line that I am not sure what it means. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 6:01pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/24 "2019-07-03T18:01:58Z")
This is the yml file for the elasticsearch node 2 ``` cluster.name: choice-cluster #node.name: corp-elk02 node.master: true node.data: true network.host: 10.101.0.141 http.port: 9300 discovery.seed_hosts: ["10.101.0.140:9300","10.101.0.141:9300"] cluster.initial_master_nodes: ["10.101.0.140:9300","10.101.0.141:9300"] xpack.security.enabled: false ``` ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 6:07pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/25 "2019-07-03T18:07:59Z")
``` [2019-07-03T18:04:13,854][INFO][o.e.n.Node] [corp-elk02] initialized [2019-07-03T18:04:13,854][INFO][o.e.n.Node] [corp-elk02] starting ... [2019-07-03T18:04:14,112][INFO][o.e.t.TransportService] [corp-elk02] publish_address {10.101.0.141:9300}, bound_addresses {10.101.0.141:9300} [2019-07-03T18:04:14,123][INFO][o.e.b.BootstrapChecks] [corp-elk02] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-07-03T18:04:14,135][INFO][o.e.c.c.Coordinator] [corp-elk02] cluster UUID [u4y2xk7mSv2x2_a8l9k2ew] [2019-07-03T18:04:14,317][INFO][o.e.c.s.MasterService] [corp-elk02] elected-as-master ([1] nodes joined)[{corp-elk02}{PpIwvyyXREKSn0HscUCmag}{kPjbrOASTQiDOAQMcomSaQ}{10.101.0.141}{10.101.0.141:9300}{ml.machine_memory=16820158464, xpack.installed=true, ml.max_open_jobs=20} elect leader, _BECOME_MASTER_TASK_, _FINISH_ELECTION_], term: 11, version: 55, reason: master node changed {previous [], current [{corp-elk02}{PpIwvyyXREKSn0HscUCmag}{kPjbrOASTQiDOAQMcomSaQ}{10.101.0.141}{10.101.0.141:9300}{ml.machine_memory=16820158464, xpack.installed=true, ml.max_open_jobs=20}]} [2019-07-03T18:04:14,886][INFO][o.e.c.s.ClusterApplierService] [corp-elk02] master node changed {previous [], current [{corp-elk02}{PpIwvyyXREKSn0HscUCmag}{kPjbrOASTQiDOAQMcomSaQ}{10.101.0.141}{10.101.0.141:9300}{ml.machine_memory=16820158464, xpack.installed=true, ml.max_open_jobs=20}]}, term: 11, version: 55, reason: Publication{term=11, version=55} [2019-07-03T18:04:15,049][WARN][o.e.b.ElasticsearchUncaughtExceptionHandler] [corp-elk02] uncaught exception in thread [main] org.elasticsearch.bootstrap.StartupException: BindHttpException[Failed to bind to [9300]]; nested: BindException[Address already in use]; ``` It pulls the kill switch after that. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 3, 2019, 6:09pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/26 "2019-07-03T18:09:19Z")
HTTP port should still be 9200. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 6:10pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/27 "2019-07-03T18:10:08Z")
Should that be the same on the other nodes to? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 3, 2019, 6:11pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/28 "2019-07-03T18:11:28Z")
Elasticsearch internally communicates on port 9300, while clients connect to port 9200 via HTTP. The seed hosts therefore look correct. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 6:22pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/29 "2019-07-03T18:22:15Z")
Okay I changed the port to 9200 and one of the nodes worked the other one now has. ``` 2019-07-03T13:19:36,620][INFO][o.e.n.Node] [choice-node1] initialized [2019-07-03T13:19:36,620][INFO][o.e.n.Node] [choice-node1] starting ... [2019-07-03T13:19:36,841][INFO][o.e.t.TransportService] [choice-node1] publish_address {10.101.0.140:9300}, bound_addresses {10.101.0.140:9300} [2019-07-03T13:19:36,877][INFO][o.e.b.BootstrapChecks] [choice-node1] bound or publishing to a non-loopback address, enforcing bootstrap checks [2019-07-03T13:19:36,912][INFO][o.e.c.c.Coordinator] [choice-node1] cluster UUID [TUPCVI9iQBK5p3Erp3L5ew] [2019-07-03T13:19:46,936][WARN][o.e.c.c.ClusterFormationFailureHelper] [choice-node1] master not discovered yet: have discovered []; discovery will continue using [10.101.0.140:9200, 10.101.0.141:9200] from hosts providers and [{choice-node1}{ukRS6dHuTiWJVjVvpSIKBQ}{5Rl6IYxaQkaeT1KAh27QZQ}{10.101.0.140}{10.101.0.140:9300}{ml.machine_memory=16820174848, xpack.installed=true, ml.max_open_jobs=20}] from last-known cluster state; node term 52, last-accepted version 96 in term 52 [2019-07-03T13:19:54,950][WARN][o.e.t.TcpTransport] [choice-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.101.0.140:59916, remoteAddress=/10.101.0.141:9200}], closing connection io.netty.handler.codec.DecoderException: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50) ``` ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 6:23pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/30 "2019-07-03T18:23:42Z")
This is part of the log I got the other part. ``` Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50) at org.elasticsearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:841) ~[elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.transport.TcpTransport.readMessageLength(TcpTransport.java:827) ~[elasticsearch-7.1.1.jar:7.1.1] at org.elasticsearch.transport.netty4.Netty4SizeHeaderFrameDecoder.decode(Netty4SizeHeaderFrameDecoder.java:40) ~[transport-netty4-client-7.1.1.jar:7.1.1] ``` ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 6:26pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/31 "2019-07-03T18:26:46Z")
I feel like this shouldn't work right? The way I am setting this up is way different from what the website says. Should this work? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 3, 2019, 6:28pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/32 "2019-07-03T18:28:18Z")
If you remove ALL port numbers it should use the defaults, which should work. It seems dome nodes still have incorrect port numbers. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 6:30pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/33 "2019-07-03T18:30:09Z")
Oh heck I changed the the port after I posted the yml my bad. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 6:36pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/34 "2019-07-03T18:36:56Z")
Should I comment out the "http.port: 9200" line or should that be left in? If so I still got the same error "master not discovered yet" ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 3, 2019, 7:05pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/35 "2019-07-03T19:05:06Z")
Yes, you should not be setting `http.port`, so delete that line. > [@thev0yager](#): > > The way I am setting this up is way different from what the website says. Which website? How are you doing something different? Most settings have sensible defaults, so the fewer settings you're changing the better. I've lost track of what state we're in now that you've fixed the port (and maybe something else). Can you share your current configs and the corresponding logs? From **both** the nodes please. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 7:10pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/36 "2019-07-03T19:10:52Z")
Absolutely! I have been looking at the elastic website on how to setup a cluster. I have pieced together what the has told me to make what I have now since what they have suggested is different from what my CTO wants me to do. Node 2 ``` cluster.name: choice-cluster #node.name: corp-elk02 node.master: true node.data: true network.host: 10.101.0.141 discovery.seed_hosts: ["10.101.0.140","10.101.0.141"] cluster.initial_master_nodes: ["10.101.0.140","10.101.0.141"] xpack.security.enabled: false ``` Node 1 ``` cluster.name: choice-cluster #node.name: corp-elk01 node.master: true node.data: true network.host: 10.101.0.140 discovery.seed_hosts: ["10.101.0.140","10.101.0.141"] cluster.initial_master_nodes: ["10.101.0.140","10.101.0.141"] xpack.security.enabled: false ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 3, 2019, 7:12pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/37 "2019-07-03T19:12:34Z")
Ok, not sure why the `node.name` lines are commented out, but otherwise this looks good. What about the logs? Again, the first couple of minutes after startup **from each node** please. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 7:15pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/38 "2019-07-03T19:15:51Z")
`[2019-07-03T19:13:40,371][INFO][o.e.c.r.a.AllocationService] [corp-elk02] Cluster health status changed from [RED] to [YELLOW] (reason: [shards started [[logstash][0]] ...]).` Okay the cluster is up now to join the other to it ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 7:16pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/39 "2019-07-03T19:16:11Z")
Do you need more logs from the working one? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 3, 2019, 7:16pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/40 "2019-07-03T19:16:48Z")
Yes, logs from both nodes please. [Previous page](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526.md?page=1) [Next page](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526.md?page=3)