# Cluster Setup 3 Node Cluster problem **URL:** \ **Category:** Elasticsearch\ **Created:** [July 2, 2019, 2:04pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526 "2019-07-02T14:04:23Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 2, 2019, 2:04pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/1 "2019-07-02T14:04:24Z")
Hi all, I have been working on getting a Elastic Stack cluster working for the past few weeks. I was told to setup the elasticsearch cluster as such. I have three nodes each one of these nodes has a elasticsearch, kibana, and logstash. I am told that I can connect a node with an E L and a K with two other nodes. The purpose for this would be redundancy. If anything goes awry on one node it will switch to the next node over. I am wondering if this is possible since the last week I have tried to get this cluster working. I have gotten the cluster running with a single node via the elasticsearch cluster setup, but I cant get any other nodes to join this said cluster. I could really use some help. If what I said is not possible I would really love to know. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 2, 2019, 3:52pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/2 "2019-07-02T15:52:26Z")
Yes, Elasticsearch normally operates as a cluster of multiple nodes for resilience and/or capacity. When you say you can't get the nodes to join up, can you share a bit more detail? What do the log messages say? A common problem with clustering is described by [this note in the docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-bootstrap-cluster.html#modules-discovery-bootstrap-cluster-joining), which also describes the solution. ---
**Author:** ![TsuWeiQuan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tsuweiquan/32/46252_2.png) [@TsuWeiQuan](https://discuss.elastic.co/u/TsuWeiQuan)\ **Post date:** [July 2, 2019, 3:55pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/3 "2019-07-02T15:55:43Z")
To get the nodes to join into 1 cluster, you must configure every node to have the same cluster name. Set the discovery.seed\_hosts & cluster.initial\_master\_nodes as seen [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/discovery-settings.html#initial_master_nodes). Once done, delete the data folder of every elasticsearch and restart them. They should all join into the same cluster. This was what i did when i was experimenting with elk. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 2, 2019, 3:57pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/4 "2019-07-02T15:57:02Z")
I am constantly getting "master node not found" I am worried that I can't connect the clusters together via elasticsearch. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 2, 2019, 3:57pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/5 "2019-07-02T15:57:50Z")
Ooo I have not done that last part yet I'll try that! ---
**Author:** ![userguy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/userguy/32/6224_2.png) [@userguy](https://discuss.elastic.co/u/userguy)\ **Post date:** [July 2, 2019, 4:17pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/6 "2019-07-02T16:17:26Z")
You need to set node.name - node.role and then clustername Also when you start a node check what is publishing address what is binding to . Ping all three servers from each other and check if you are reachable . Check if selinux and firewall settings are tuned . Also can you paste your cluster configuration file if above steps does not work ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 2, 2019, 4:25pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/7 "2019-07-02T16:25:10Z")
> [@thev0yager](#): > > I am constantly getting "master node not found" I am worried that I can't connect the clusters together via elasticsearch. I do not think that "master node not found" is a message that Elasticsearch would emit. Please share logs and error messages and so on exactly as they come from Elasticsearch, including surrounding context. It may not look relevant, but it's better to have more detail than less. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 2, 2019, 4:33pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/8 "2019-07-02T16:33:11Z")
[2019-07-02T11:29:42,716][WARN][o.e.c.c.ClusterFormationFailureHelper] [choice-node1] master not discovered yet: have discovered ; discovery will continue using ~~~[10.101.0.140:9200, 10.101.0.141:9200] from hosts providers and [{choice-node1}{ukRS6dHuTiWJVjVvpSIKBQ}{eqCAFzP9QPGkDd8lfDgZXA}{10.101.0.140}{10.101.0.140:9300}{ml.machine\_memory=16820174848, xpack.installed=true, ml.max\_open\_jobs=20}] from last-known cluster state; node term 52, last-accepted version 96 in term 52~~~ ---
**Author:** ![userguy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/userguy/32/6224_2.png) [@userguy](https://discuss.elastic.co/u/userguy)\ **Post date:** [July 2, 2019, 4:37pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/9 "2019-07-02T16:37:00Z")
it says it is not able to communicate with the server mentioned in hosts .check if they are able to communicate with each other using ping and telnet on port 9200 and 9300 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 3, 2019, 5:01am UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/10 "2019-07-03T05:01:11Z")
Is this the only message on your logs? I think there will also be some error messages, including stack traces, explaining the issue, although you might have to wait a few minutes for them to appear. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 1:20pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/11 "2019-07-03T13:20:41Z")
Okay I checked both of those ports 9300 isn't even listed, but 9200 has multiple working connections. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 1:21pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/12 "2019-07-03T13:21:26Z")
I will keep an eye on anymore errors that come through so far it has just been the same error over and over again. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 3, 2019, 1:47pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/13 "2019-07-03T13:47:54Z")
I think your problem is that you're using port 9200 in `discovery.seed_hosts` rather than 9300, but I am surprised that this is not yielding more error messages. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 3:35pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/14 "2019-07-03T15:35:35Z")
It just gave me this one. ``` Caused by: java.io.StreamCorruptedException: invalid internal transport message format, got (48,54,54,50) at org.elasticsearch.transport.TcpTransport.readHeaderBuffer(TcpTransport.java:841) ~[elasticsearch-7.1.1.jar:7.1.1] ``` ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 3, 2019, 4:12pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/15 "2019-07-03T16:12:56Z")
It's always a good idea to share the whole stack trace. It's normally quite hard to help from just a line or two. In this case, that error is consistent with [what I said above](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/13). ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 4:45pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/16 "2019-07-03T16:45:39Z")
Would you suggest me continuing this path of a cluster of nodes each node running all three of the services or would you suggest doing it the way the elastic website says I should have this setup? As in each node has one service and I make the cluster out of that. ---
**Author:** ![elasticforme](https://avatars.discourse-cdn.com/v4/letter/e/f05b48/32.png) [@elasticforme](https://discuss.elastic.co/u/elasticforme)\ **Post date:** [July 3, 2019, 4:53pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/17 "2019-07-03T16:53:53Z")
@thev0yager as david said if you post your config file elasticsearch.yml and more log entry someone will be able to help you. This seems simple config problem. ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 4:56pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/18 "2019-07-03T16:56:52Z")
``` cluster.name: choice-cluster #node.name: corp-elk02 node.master: true node.data: true network.host: 10.101.0.141 http.port: 9300 discovery.seed_hosts: ["10.101.0.140:9300","10.101.0.141:9300"] cluster.initial_master_nodes: ["10.101.0.140:9300","10.101.0.141:9300"] xpack.security.enabled: false ``` ---
**Author:** ![thev0yager](https://avatars.discourse-cdn.com/v4/letter/t/c5a1d2/32.png) [@thev0yager](https://discuss.elastic.co/u/thev0yager)\ **Post date:** [July 3, 2019, 5:00pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/19 "2019-07-03T17:00:39Z")
My other node has an identical yml file accept the node name is different. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 3, 2019, 5:35pm UTC](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526/20 "2019-07-03T17:35:08Z")
This is inconsistent with the messages you shared above. The message said this: `... discovery will continue using [10.101.0.140:9200, 10.101.0.141:9200] from hosts providers...` however your config now says this: ```auto discovery.seed_hosts: ["10.101.0.140:9300","10.101.0.141:9300"] ``` Note the different port numbers. If you've updated your config you should now be getting different messages. Did you update your config? What is Elasticsearch saying now? Can you share all the logs emitted for the first couple of minutes after startup, from all of your nodes? Also your original post was about three nodes, but you seem to be talking about two nodes now. How many nodes are there? [Next page](https://discuss.elastic.co/t/cluster-setup-3-node-cluster-problem/188526.md?page=2)