# Elasticsearch 7 nodes cluster issues **URL:** \ **Category:** Elasticsearch\ **Created:** [June 19, 2017, 12:05pm UTC](https://discuss.elastic.co/t/elasticsearch-7-nodes-cluster-issues/89923 "2017-06-19T12:05:23Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![baravit](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@baravit](https://discuss.elastic.co/u/baravit)\ **Post date:** [June 19, 2017, 12:05pm UTC](https://discuss.elastic.co/t/elasticsearch-7-nodes-cluster-issues/89923/1 "2017-06-19T12:05:23Z")
Hey guys, I'm making some changes in my elasticsearch cluster and need a little help with the nodes role allocation and the instances configuration. till now we had 3 nodes working as data and master-eligible together and kibana on another node listening to the cluster. This setup gave us really hard time and we decided to take elasticsearch one step ahead. Our Data: the cluster managed with curator crons that keeps 2 weeks of raw-data (time-based indices) open for debug querying and one more week closed for emergencies. besides that we have another index that kept update from the raw-data streaming. The cluster holds ~100GB of data in 15 different indices. each index divided to 5 shards with 1 replica. The current setup composed from 7 nodes 3 dedicated master nodes - 2cpu 4gb ram. 3 dedicated data nodes - 4cpu 16gb ram. 1 coordinate node with kibana - 4cpu 8gb ram. Those are my questions: 1.The instances setup makes sense? reasonable? I read that master nodes can usually be quite "light" compared to data nodes, is 2gb ram instance with 1gb heap sounds good? 2.when i get /\_cat/nodes stat i notice that ram.precent is pretty high (above 90 on all nodes): ip heap.percent ram.percent cpu load\_1m load\_5m load\_15m node.role master name 172.31.17.103 23 99 11 5.88 5.22 5.05 d - DATA1 172.31.12.76 31 94 37 0.30 0.20 0.12 m \* MASTER1 172.31.29.43 22 99 7 5.26 5.00 4.98 d - DATA3 172.31.14.236 12 89 0 0.00 0.00 0.00 m - MASTER2 172.31.14.54 21 97 1 0.12 0.04 0.01 - - KIBANA 172.31.14.55 23 93 0 0.00 0.00 0.00 m - MASTER3 172.31.17.46 20 99 9 5.88 5.12 4.97 d - DATA2 this is a proper state? or should i limit this config setting? how can it be done? 3.our system handles with ~150 docs per sec but we want to be able to scale up to 1000 (with data increasing to ~500gb) with the minimal adjustments in the future. any recommendations that will get us closer to that spot? 4.should we add dedicated coordinate node besides the node with kibana? what the immediate effects of such change? for now it's like we ran the cluster without any coordinate node because the node with kibana running elasticsearch on localhost and communicate with the cluster through the transport client. 1. how should we talk with the cluster from the web client? provide list of all the nodes ip? only the masters? 2. does cluster of 2 data nodes with 32ram each sounds better then the current setup? 3. separate monitoring cluster - mandatory configuration on production env? besides the monitor consistency it takes some of the cluster load? Thanks so much for your help:) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 19, 2017, 12:30pm UTC](https://discuss.elastic.co/t/elasticsearch-7-nodes-cluster-issues/89923/2 "2017-06-19T12:30:40Z")
It sounds to me like you may be having far too many small shards. 15 indices , each with 10 shards per day, gives 2100 shards over a 14 day period. Given the data volumes you have mentioned this seems excessive as each shard has overhead and consumes resources. The easiest way to reduce the number of shards would be to reduce the number of primary shards per index from 5 to 1. Aim for an average shard size between a few Gb and a few tens of GB. If you are on a recent release of Elasticsearch you can use the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/5.4/indices-shrink-index.html), but otherwise you may need implement the change and wait for data to be phased out in order to see the effect. ---
**Author:** ![baravit](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@baravit](https://discuss.elastic.co/u/baravit)\ **Post date:** [June 19, 2017, 12:55pm UTC](https://discuss.elastic.co/t/elasticsearch-7-nodes-cluster-issues/89923/3 "2017-06-19T12:55:16Z")
Hey Christian\_Dahlqvist, Tnx for your response.. the shards allocation was carried out with a view to future demand to support total data volume of 0.5T - 1T. at the time, it seems like the right choice for this kind of setup is 5 shards and 1 replica. I'm going to change the future indices to create with 1 shard and monitor the changes. Thank you. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 19, 2017, 12:58pm UTC](https://discuss.elastic.co/t/elasticsearch-7-nodes-cluster-issues/89923/4 "2017-06-19T12:58:50Z")
The good thing about time-based indices is that you can easily change the number of shards for the next day and therefore adapt as volumes grow. Is that data volume for the entire cluster or the amount of data indexed per day? ---
**Author:** ![baravit](https://avatars.discourse-cdn.com/v4/letter/b/b2d939/32.png) [@baravit](https://discuss.elastic.co/u/baravit)\ **Post date:** [June 19, 2017, 1:01pm UTC](https://discuss.elastic.co/t/elasticsearch-7-nodes-cluster-issues/89923/5 "2017-06-19T13:01:56Z")
yea, i'm going to change the template so that tomorrow index will create with 1 shard and i will share the results with u ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 17, 2017, 1:02pm UTC](https://discuss.elastic.co/t/elasticsearch-7-nodes-cluster-issues/89923/6 "2017-07-17T13:02:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.