# Elasticsearch cluster is not working **URL:** \ **Category:** Elasticsearch\ **Created:** [November 15, 2018, 12:28pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853 "2018-11-15T12:28:58Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![jatinseth2007](https://avatars.discourse-cdn.com/v4/letter/j/b19c9b/32.png) [@jatinseth2007](https://discuss.elastic.co/u/jatinseth2007)\ **Post date:** [November 15, 2018, 12:28pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/1 "2018-11-15T12:28:58Z")
I am trying to run `ES(6.4.3) cluster` where I have 2 nodes and I want to make one node as master and data and other as only data. My both nodes are up and running but not able to communicate with each other to discover a cluster. If I run `telnet` command to check then I am able to run telnet to `localhost` on `port 9300` but not able to connect to other node. I am using `Digital ocean` and `ubuntu 16.04` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 16, 2018, 12:04am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/2 "2018-11-16T00:04:07Z")
> [@jatinseth2007](#): > > I am trying to run `ES(6.4.3) cluster` where I have 2 nodes and I want to make one node as master and data and other as only data. That's less than ideal, see [Important Configuration Changes | Elasticsearch: The Definitive Guide [2.x] | Elastic](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/important-configuration-changes.html#_minimum_master_nodes). > [@jatinseth2007](#): > > My both nodes are up and running but not able to communicate with each other to discover a cluster. You will need to show your config and your logs for us to really help. Please format your code/config/logs using the `` button, or markdown style back ticks. It helps to make things easy to read which helps us help you 🙂 ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [November 16, 2018, 4:11am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/3 "2018-11-16T04:11:07Z")
@jatinseth2007, port 9300 should be open on both the nodes. Please check the elasticsearch logs for more info. Thanks. ---
**Author:** ![jatinseth2007](https://avatars.discourse-cdn.com/v4/letter/j/b19c9b/32.png) [@jatinseth2007](https://discuss.elastic.co/u/jatinseth2007)\ **Post date:** [November 16, 2018, 7:24am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/4 "2018-11-16T07:24:20Z")
Config for one server: ``` # ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name : campaygn-production # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: campaygn-production-elasticsearch-node-0 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /var/lib/elasticsearch # # Path to log files: # path.logs: /var/log/elasticsearch # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 209.97.184.107 # # Set a custom port for HTTP: # http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.zen.ping.unicast.hosts: ["209.97.184.107","142.93.42.161"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true #script.inline: true indices.query.bool.max_clause_count: 100000 #path.repo: ["/tmp/backups/elasticsearch/elasticsearch"] #define node 1 as master-eligible: node.master: true #define nodes 1 as data node: node.data: true ``` and logs for same server: ``` [2018-03-16T10:57:38,250][INFO][o.e.t.TransportService] [lhD23sr] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} [2018-03-16T10:57:41,309][INFO][o.e.c.s.MasterService] [lhD23sr] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {lhD23sr}{lhD23srPT3Wb0Kug2XmkiA}{1VfK_uzZRayGdEX4sABdyQ}{127.0.0.1}{127.0.0.1:9300} [2018-03-16T10:57:41,314][INFO][o.e.c.s.ClusterApplierService] [lhD23sr] new_master {lhD23sr}{lhD23srPT3Wb0Kug2XmkiA}{1VfK_uzZRayGdEX4sABdyQ}{127.0.0.1}{127.0.0.1:9300}, reason: apply cluster state (from master [master {lhD23sr}{lhD23srPT3Wb0Kug2XmkiA}{1VfK_uzZRayGdEX4sABdyQ}{127.0.0.1}{127.0.0.1:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]]) [2018-03-16T10:57:41,341][INFO][o.e.h.n.Netty4HttpServerTransport] [lhD23sr] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200} [2018-03-16T10:57:41,341][INFO][o.e.n.Node] [lhD23sr] started [2018-03-16T10:57:41,347][INFO][o.e.g.GatewayService] [lhD23sr] recovered [0] indices into cluster_state [2018-03-16T11:07:58,242][INFO][o.e.n.Node] [lhD23sr] stopping ... [2018-03-16T11:07:58,270][INFO][o.e.n.Node] [lhD23sr] stopped [2018-03-16T11:07:58,270][INFO][o.e.n.Node] [lhD23sr] closing ... [2018-03-16T11:07:58,278][INFO][o.e.n.Node] [lhD23sr] closed ``` ---
**Author:** ![jatinseth2007](https://avatars.discourse-cdn.com/v4/letter/j/b19c9b/32.png) [@jatinseth2007](https://discuss.elastic.co/u/jatinseth2007)\ **Post date:** [November 16, 2018, 7:26am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/5 "2018-11-16T07:26:59Z")
Config for 2nd server : ``` # ======================== Elasticsearch Configuration ========================= # # NOTE: Elasticsearch comes with reasonable defaults for most settings. # Before you set out to tweak and tune the configuration, make sure you # understand what are you trying to accomplish and the consequences. # # The primary way of configuring a node is via this file. This template lists # the most important settings you may want to configure for a production cluster. # # Please consult the documentation for further information on configuration options: # https://www.elastic.co/guide/en/elasticsearch/reference/index.html # # ---------------------------------- Cluster ----------------------------------- # # Use a descriptive name for your cluster: # cluster.name: campaygn-production # # ------------------------------------ Node ------------------------------------ # # Use a descriptive name for the node: # node.name: campaygn-production-elasticsearch-node-1 # # Add custom attributes to the node: # #node.attr.rack: r1 # # ----------------------------------- Paths ------------------------------------ # # Path to directory where to store the data (separate multiple locations by comma): # path.data: /var/lib/elasticsearch # # Path to log files: # path.logs: /var/log/elasticsearch # # ----------------------------------- Memory ----------------------------------- # # Lock the memory on startup: # #bootstrap.memory_lock: true # # Make sure that the heap size is set to about half the memory available # on the system and that the owner of the process is allowed to use this # limit. # # Elasticsearch performs poorly when the system is swapping the memory. # # ---------------------------------- Network ----------------------------------- # # Set the bind address to a specific IP (IPv4 or IPv6): # network.host: 142.93.42.161 # # Set a custom port for HTTP: # http.port: 9200 # # For more information, consult the network module documentation. # # --------------------------------- Discovery ---------------------------------- # # Pass an initial list of hosts to perform discovery when new node is started: # The default list of hosts is ["127.0.0.1", "[::1]"] # discovery.zen.ping.unicast.hosts: ["209.97.184.107","142.93.42.161"] # # Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1): # #discovery.zen.minimum_master_nodes: # # For more information, consult the zen discovery module documentation. # # ---------------------------------- Gateway ----------------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: # #gateway.recover_after_nodes: 3 # # For more information, consult the gateway module documentation. # # ---------------------------------- Various ----------------------------------- # # Require explicit names when deleting indices: # #action.destructive_requires_name: true indices.query.bool.max_clause_count: 100000 #define node 1 as master-eligible: node.master: false #define nodes 1 as data node: node.data: true ``` And logs for 2nd server: ``` [2018-11-14T10:46:57,186][INFO][o.e.n.Node] [q3IrN9m] starting ... [2018-11-14T10:46:57,354][INFO][o.e.t.TransportService] [q3IrN9m] publish_address {127.0.0.1:9300}, bound_addresses {[::1]:9300}, {127.0.0.1:9300} [2018-11-14T10:47:00,413][INFO][o.e.c.s.ClusterService] [q3IrN9m] new_master {q3IrN9m}{q3IrN9mDQrS8TkwchxdqUw}{o9D1RMJBTdWXBEzJlvF-nQ}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined) [2018-11-14T10:47:00,448][INFO][o.e.h.n.Netty4HttpServerTransport] [q3IrN9m] publish_address {127.0.0.1:9200}, bound_addresses {[::1]:9200}, {127.0.0.1:9200} [2018-11-14T10:47:00,448][INFO][o.e.n.Node] [q3IrN9m] started [2018-11-14T10:47:00,450][INFO][o.e.g.GatewayService] [q3IrN9m] recovered [0] indices into cluster_state [2018-11-14T10:57:50,942][INFO][o.e.n.Node] [q3IrN9m] stopping ... [2018-11-14T10:57:50,965][INFO][o.e.n.Node] [q3IrN9m] stopped [2018-11-14T10:57:50,965][INFO][o.e.n.Node] [q3IrN9m] closing ... [2018-11-14T10:57:50,974][INFO][o.e.n.Node] [q3IrN9m] closed ``` ---
**Author:** ![jatinseth2007](https://avatars.discourse-cdn.com/v4/letter/j/b19c9b/32.png) [@jatinseth2007](https://discuss.elastic.co/u/jatinseth2007)\ **Post date:** [November 16, 2018, 7:28am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/6 "2018-11-16T07:28:48Z")
Regarding [https://www.elastic.co/guide/en/elasticsearch/guide/2.x/important-configuration-changes.html#\_minimum\_master\_nodes](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/important-configuration-changes.html#_minimum_master_nodes). I am aware of the guidelines and will add one more once this will be successful. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 16, 2018, 7:29am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/7 "2018-11-16T07:29:01Z")
> [@jatinseth2007](#): > > [2018-11-14T10:46:57,354][INFO][o.e.t.TransportService] [q3IrN9m] publish\_address {127.0.0.1:9300}, bound\_addresses {[::1]:9300}, {127.0.0.1:9300} Both are still only binding to `localhost` for some reason. However you shouldn't really use a public IP for your cluster, unless you have some kind of access control managing things. ---
**Author:** ![jatinseth2007](https://avatars.discourse-cdn.com/v4/letter/j/b19c9b/32.png) [@jatinseth2007](https://discuss.elastic.co/u/jatinseth2007)\ **Post date:** [November 16, 2018, 7:30am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/8 "2018-11-16T07:30:39Z")
Regarding ports my ufw status is given below: for one server Status: active ``` To Action From -- ------ ---- OpenSSH ALLOW Anywhere Nginx Full ALLOW Anywhere 3306 ALLOW Anywhere 9300 ALLOW Anywhere OpenSSH (v6) ALLOW Anywhere (v6) Nginx Full (v6) ALLOW Anywhere (v6) 3306 (v6) ALLOW Anywhere (v6) 9300 (v6) ALLOW Anywhere (v6) ``` for 2nd server ``` Status: active To Action From -- ------ ---- 9300 ALLOW Anywhere 9200 ALLOW Anywhere 22 ALLOW Anywhere 80 ALLOW Anywhere 23/tcp ALLOW Anywhere 9300 (v6) ALLOW Anywhere (v6) 9200 (v6) ALLOW Anywhere (v6) 22 (v6) ALLOW Anywhere (v6) 80 (v6) ALLOW Anywhere (v6) 23/tcp (v6) ALLOW Anywhere (v6) ``` ---
**Author:** ![jatinseth2007](https://avatars.discourse-cdn.com/v4/letter/j/b19c9b/32.png) [@jatinseth2007](https://discuss.elastic.co/u/jatinseth2007)\ **Post date:** [November 16, 2018, 7:32am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/9 "2018-11-16T07:32:31Z")
I can use private IP too, once it will start working. So, what I should put as a setting ? I have given network host as mentioned in one tutorial on digital ocean. When I am trying to send request from second server curl -XGET '[http://142.93.42.161:9200/\_cluster/state?pretty](http://142.93.42.161:9200/_cluster/state?pretty)' following error is coming. ``` { "error" : { "root_cause" : [ { "type" : "master_not_discovered_exception", "reason" : null } ], "type" : "master_not_discovered_exception", "reason" : null }, "status" : 503 } ``` ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [November 16, 2018, 9:51am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/10 "2018-11-16T09:51:21Z")
@jatinseth2007, Can you please define the below setting in your master node in `elasticsearch.yml` file and restart the elasticsearch service: ```auto node.master: true node.data: false ``` just now read your configuration you have defined above config. Can you please define below setting into `elasticsearch.yml` ```auto discovery.zen.minimum_master_nodes: 2 ``` Also please provide the output of below command from both servers: ```auto netstat -tunlp ``` Thanks. ---
**Author:** ![jatinseth2007](https://avatars.discourse-cdn.com/v4/letter/j/b19c9b/32.png) [@jatinseth2007](https://discuss.elastic.co/u/jatinseth2007)\ **Post date:** [November 22, 2018, 11:52am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/11 "2018-11-22T11:52:13Z")
I found out the problem and you were correct, it was permission issue but at server the permissions were fine but for Digital Ocean firewall they have one extra firewall and if you don't give any permission then by default it's not allowed. So, I had to give permissions specifically for those ports to each server. Just one question I have is how to make sure cluster is working good, I have run following curl request and response is also given: ``` curl -XGET -H "Content-Type: application/json" http://localhost:9200/_cluster/health?pretty=true { "cluster_name" : "production", "status" : "yellow", "timed_out" : false, "number_of_nodes" : 2, "number_of_data_nodes" : 2, "active_primary_shards" : 10, "active_shards" : 11, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 9, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 55.00000000000001 } ``` ---
**Author:** ![Tek\_Chand](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tek_chand/32/34318_2.png) [@Tek\_Chand](https://discuss.elastic.co/u/Tek_Chand)\ **Post date:** [November 23, 2018, 4:18am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/12 "2018-11-23T04:18:55Z")
@jatinseth2007, > [@jatinseth2007](#): > > I found out the problem Glad to hear that. 🙂 > [@jatinseth2007](#): > > Just one question I have is how to make sure cluster is working good, If you see the output of your curl command, the cluster health is showing `yellow`. It should be `green`. Currently its showing status `yellow` because there are 9 shards which are unassigned. The reason for unassigned shards is, you have only one data node. So your primary shard and replica shards are on same data node. So you need at least 2 data node. So your primary shard and replica shards can be on different data node. Then the cluster status will also changed and it will become `green`. Let say, you have 5 primary shards (P1,P2....p5) and 1 replica for each primary shards. Then you will have total 10 shards 5 primary and 5 replicas i.e R1,R2,...R5 (one for each primary shards). If you have 2 data node then your first data node may have P1,P2,R3,R4,R5 and second data node may have P3,P4,P5,R1,R2. It managed by Elasticsearch itself. Hope so above point will help you. Kindly let me know if you any other question. Thanks. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [December 21, 2018, 4:19am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-is-not-working/156853/13 "2018-12-21T04:19:02Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.