# Elasticsearch cluster Query **URL:** \ **Category:** Elasticsearch\ **Created:** [September 9, 2017, 10:37am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929 "2017-09-09T10:37:21Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![rrs](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rrs](https://discuss.elastic.co/u/rrs)\ **Post date:** [September 9, 2017, 10:37am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/1 "2017-09-09T10:37:21Z")
Hello, We are in process of transferring logs over Elasticsearch will get 100GB data a day with 30 days retention policy. As per my understanding Elasticsearch layer requires more resources than Logstash & Kibana. We deciced provisioning two VM's with 8Core & 8GB RAM for both Logstash & Kibana and 3 VM's with same resources for Elasticsearch. We don't have enough local storage to keep processed elasticserach data, hence we decided allocating nfs storage with 3TB. So, my question is 3TB volume will be mounted across all three elasticsearch nodes or 3\*1 TB volumes get mounted on three nodes. Please suggest how my cluster will be equipped. Thanks in advance. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 9, 2017, 9:32pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/2 "2017-09-09T21:32:29Z")
We do not recommend using NFS as a direct store for Elasticsearch to run with. Even more so if you use it as a single shared volume, your performance will be horrible. ---
**Author:** ![rrs](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rrs](https://discuss.elastic.co/u/rrs)\ **Post date:** [September 15, 2017, 7:26am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/3 "2017-09-15T07:26:47Z")
Hi Mark, Thanks for quick help on this thread. For now, we going with SAN storage with 1TB on each node. As far as I know the beauty of Elasticsearch is it caters High Availability. As per that feature configured our Elasticsearch nodes as below that uses "split brain". Can you correct me, if I am wrong. prd-els-1: node.master: true node.data: true discovery.zen.ping.unicast.hosts: ["prd-els-1","prd-els-2","prd-els-3"] discovery.zen.minimum\_master\_nodes: 2 prd-els-2: node.master: true node.data: true discovery.zen.ping.unicast.hosts: ["prd-els-1","prd-els-2","prd-els-3"] discovery.zen.minimum\_master\_nodes: 2 prd-els-3: node.master: false node.data: true discovery.zen.ping.unicast.hosts: ["prd-els-1","prd-els-2","prd-els-3"] discovery.zen.minimum\_master\_nodes: 2 ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 15, 2017, 8:18am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/4 "2017-09-15T08:18:32Z")
Set `node.master` to `true` for the third node as well. This will allow a master to be elected even if you lose one node, and will make the cluster more resilient. Otherwise it looks good. ---
**Author:** ![rrs](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rrs](https://discuss.elastic.co/u/rrs)\ **Post date:** [September 15, 2017, 11:55am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/5 "2017-09-15T11:55:34Z")
Hi, Node of the configurations work to make Elasticserach up and running. I am seeing below errors, if we make 2 master + 1 non-master nodes and 3 master nodes as well. I also verified telnet is able to connect to both ports 9200/9300 from each host and its pair hosts. The version we used elasticsearch-5.5.2-1.noarch installed on Centos-7.2. Can you please suggest. [2017-09-15T11:47:19,491][INFO][o.e.d.z.ZenDiscovery] [prd-els-1] failed to send join request to master [{prd-els-3}{DQHiqOENQpWkKlb7WYAC6g}{KqE\_1d7YQfaKpV2Et2kcVA}{prd-els-3}{[10.xx.xxx.xxx:9300](http://10.xx.xxx.xxx:9300)}], reason [RemoteTransportException[[prd-els-3][[10.xx.xxx.xxx:9300](http://10.xx.xxx.xxx:9300)][internal:discovery/zen/join]]; nested: NotMasterException[Node [{prd-els-3}{DQHiqOENQpWkKlb7WYAC6g}{KqE\_1d7YQfaKpV2Et2kcVA}{prd-els-3}{[10.xx.xxx.xxx:9300](http://10.xx.xxx.xxx:9300)}] not master for join request]; ], tried [3] times ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 15, 2017, 12:04pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/6 "2017-09-15T12:04:41Z")
Is Elasticsearch starting up successfully on all nodes? ---
**Author:** ![rrs](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rrs](https://discuss.elastic.co/u/rrs)\ **Post date:** [September 15, 2017, 12:10pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/7 "2017-09-15T12:10:58Z")
Hi, Yes, its getting started for every restart. The log output it generated is [2017-09-15T12:06:45,624][INFO][o.e.n.Node] [prd-els-1] initializing ... [2017-09-15T12:06:45,696][INFO][o.e.e.NodeEnvironment] [prd-els-1] using [1] data paths, mounts [[/ (rootfs)]], net usable\_space [9gb], net total\_space [11.6gb], spins? [unknown], types [rootfs] [2017-09-15T12:06:45,696][INFO][o.e.e.NodeEnvironment] [prd-els-1] heap size [2.9gb], compressed ordinary object pointers [true] [2017-09-15T12:06:45,703][INFO][o.e.n.Node] [prd-els-1] node name [prd-els-1], node ID [DQHiqOENQpWkKlb7WYAC6g] [2017-09-15T12:06:45,704][INFO][o.e.n.Node] [prd-els-1] version[5.5.2], pid[6594], build[b2f0c09/2017-08-14T12:33:14.154Z], OS[Linux/3.10.0-327.18.2.el7.x86\_64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0\_141/25.141-b16] [2017-09-15T12:06:45,704][INFO][o.e.n.Node] [prd-els-1] JVM arguments [-Xms3g, -Xmx3g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/usr/share/elasticsearch] [2017-09-15T12:06:46,563][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [aggs-matrix-stats] [2017-09-15T12:06:46,563][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [ingest-common] [2017-09-15T12:06:46,563][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [lang-expression] [2017-09-15T12:06:46,563][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [lang-groovy] [2017-09-15T12:06:46,563][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [lang-mustache] [2017-09-15T12:06:46,563][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [lang-painless] [2017-09-15T12:06:46,563][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [parent-join] [2017-09-15T12:06:46,563][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [percolator] [2017-09-15T12:06:46,564][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [reindex] [2017-09-15T12:06:46,564][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [transport-netty3] [2017-09-15T12:06:46,564][INFO][o.e.p.PluginsService] [prd-els-1] loaded module [transport-netty4] [2017-09-15T12:06:46,564][INFO][o.e.p.PluginsService] [prd-els-1] no plugins loaded [2017-09-15T12:06:47,941][INFO][o.e.d.DiscoveryModule] [prd-els-1] using discovery type [zen] [2017-09-15T12:06:48,468][INFO][o.e.n.Node] [prd-els-1] initialized [2017-09-15T12:06:48,468][INFO][o.e.n.Node] [prd-els-1] starting ... [2017-09-15T12:06:48,621][INFO][o.e.t.TransportService] [prd-els-1] publish\_address {[10.xx.xxx.xxx:9300](http://10.xx.xxx.xxx:9300)}, bound\_addresses {[10.xx.xxx.xxx:9300](http://10.xx.xxx.xxx:9300)} [2017-09-15T12:06:48,633][INFO][o.e.b.BootstrapChecks] [prd-els-1] bound or publishing to a non-loopback or non-link-local address, enforcing bootstrap checks [2017-09-15T12:06:52,052][INFO][o.e.d.z.ZenDiscovery] [prd-els-1] failed to send join request to master [{prd-els-3}{DQHiqOENQpWkKlb7WYAC6g}{KqE\_1d7YQfaKpV2Et2kcVA}{prd-els-3}{[10.xx.xxx.xxx:9300](http://10.xx.xxx.xxx:9300)}], reason [RemoteTransportException[[prd-els-3][[10.xx.xxx.xxx:9300](http://10.xx.xxx.xxx:9300)][internal:discovery/zen/join]]; nested: NotMasterException[Node [{prd-els-3}{DQHiqOENQpWkKlb7WYAC6g}{KqE\_1d7YQfaKpV2Et2kcVA}{prd-els-3}{[10.xx.xxx.xxx:9300](http://10.xx.xxx.xxx:9300)}] not master for join request]; ], tried [3] times ---
**Author:** ![rrs](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rrs](https://discuss.elastic.co/u/rrs)\ **Post date:** [September 15, 2017, 12:17pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/8 "2017-09-15T12:17:56Z")
Hi, And, the configuration we made is as below and ensured elasticserach has proper privileges to write logs/indexes. One of the master node: [cluster.name](http://cluster.name): prod\_app\_01 [node.name](http://node.name): prd-els-2 node.master: true node.data: true path.data: /lsh/data path.logs: /lsh/logs network.host: 10.xx.xxx.102 http.port: 9200 discovery.zen.ping.unicast.hosts: ["prd-els-1", "prd-els-2", "prd-els-3"] discovery.zen.minimum\_master\_nodes: 2 network.publish\_host: prd-els-2 One of the data\_node: [cluster.name](http://cluster.name): prod\_app\_01 [node.name](http://node.name): prd-els-3 node.master: true node.data: true path.data: /lsh/data path.logs: /lsh/logs network.host: 10.xx.xxx.103 http.port: 9200 discovery.zen.ping.unicast.hosts: ["prd-els-1", "prd-els-2", "prd-els-3"] discovery.zen.minimum\_master\_nodes: 2 network.publish\_host: prd-els-3 ---
**Author:** ![rrs](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rrs](https://discuss.elastic.co/u/rrs)\ **Post date:** [September 15, 2017, 2:54pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/9 "2017-09-15T14:54:28Z")
Hi Christian\_Dahlqvist / Mark, Thanks for your support. I have fixed the issue now cluster status looks pretty good. The RCA is that since my VMWare supports cloning, first I installed Elasticserach one one VM with required parameters then I cloned same VM to another two nodes. As I know Elasticsearch uses ID's the same ID is replicated for other two nodes while cloning, which causes Elasticsearch to see only one ID even though there are 3 nodes. I realized and re-installed/re-configured elasticserach. Now, it works as expected. ~]# curl -s 'prd-els-1:9200/\_cat/nodes' 10.120.181.129 11 85 6 0.34 0.13 0.08 mdi - prd-els-1 10.120.181.131 16 53 3 0.06 0.06 0.05 mdi \* prd-els-2 10.120.181.132 11 78 2 0.05 0.07 0.06 di - prd-els-3 One last question is we have Logstash layer installed on another VM where logstash forwards processed logs to prd-els-1 which is one of my master that distributes indexes across other nodes. If my prd-els-1 goes down, how Logstash sends the new logs? And, my kibana config points to primary master prd-els-1. How can I search logs? Please advice. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 15, 2017, 2:59pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/10 "2017-09-15T14:59:11Z")
You should always aim to have 3 master-eligible nodes and set `minimum_master_nodes` to `2`. Logstash should have all 3 nodes defined so that it can distribute load across the cluster. ---
**Author:** ![rrs](https://avatars.discourse-cdn.com/v4/letter/r/d9b06d/32.png) [@rrs](https://discuss.elastic.co/u/rrs)\ **Post date:** [September 15, 2017, 3:12pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/11 "2017-09-15T15:12:26Z")
Hi, It makes sense. We follow it to make all 3 nodes as master-eligible. Due to resources crunch and this is initial setup the cluster will act as making indexes and serving seach traffic as well. Could you also suggest best how it can be fulfilled with existing 3 node cluster and below points If my prd-els-1 goes down, how Logstash sends the new logs? And, my kibana config points to primary master prd-els-1. How can I search logs? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 13, 2017, 3:12pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-query/99929/12 "2017-10-13T15:12:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.