# Elasticsearch cluster scaling from 3 nodes to 5 nodes to open many indices/shards, best practices

**URL:** <https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988>\
**Category:** Elasticsearch\
**Created:** [January 21, 2019, 7:10am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988 "2019-01-21T07:10:01Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![kishore419](https://avatars.discourse-cdn.com/v4/letter/k/b4bc9f/32.png) [@kishore419](https://discuss.elastic.co/u/kishore419)\
**Post date:** [January 21, 2019, 7:10am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988/1 "2019-01-21T07:10:01Z")

</div>

We are using three node Elasticsearch cluster in AWS Cloud.  
here are the configurations.  
Instance Type: M4 4X Large (Three nodes), each node configurations are given below.  
CPU, 16 cores, RAM 64GB, EBS volumes: 4TB \* 4 volumes  
EBS Volume Utilizations: 2.5Tb \* 4 volumes.  
JVM heapsapce : 31.5GB, virtual memory : 10GB

## With the above configuration we could able to 'open 520 indices active\_primary\_shards: 2800' from 6300 indexes (Appx 39000 shards). JVM utlization details are below:

ip heap.percent ram.percent cpu load\_1m load\_5m load\_15m  
IP1 63 99 5 0.60 0.46 0.45mdi  
IP2 37 98 4 0.45 0.34 0.32mdi  
IP3 56 96 4 0.25 0.19 0.25mdi

Master node: \* and node is : node-IP1

## Requirement:

we have to open 1750 indices (Appx 11600 shards) in the above cluster to meet my customers requirement.  
Unfortunately we don't have sufficient Staging stack to test the following proposals, hence please look the below proposals and suggest me on best approach/possible precautions.

Proposal 1:  
We would like to join/add 2 more nodes i.e. M4 4X Large nodes to the above cluster (make 5 node cluster).  
-\> We will be configured JVM heap space as 31.5GB similar to the other nodes.  
-\> Two new nodes would be considered as only Master nodes (data node flag - false)  
-\> No EBS volumes for new nodes, since existing EBS volume size is sufficient for my requirement  
-\> minimum master nodes 3 and data nodes 2

JVM heap space would be increased with the above proposal and we could open atlready 1500+ indices with above setup.  
please share your thoughts

Proposal 2:  
We would like to join/add 2 more nodes i.e. M4 4X Large nodes to the above cluster (make 5 node cluster).  
-\> We will be configured JVM heap space as 31.5GB similar to the other nodes.  
-\> Two new nodes would be considered as Master and data nodes  
-\> Only one EBS volume per node and capacity is 10TB.  
-\> minimum master nodes 3 and data nodes 2

JVM heap space would be increased with the above proposal and we could open already 1500+ indices with above setup.  
please share your thoughts

Proposal 3:  
Configuration is similar to the existing cluster,  
minimum master nodes 3 and data nodes 2

JVM heap space would be increased with the above proposal and we could open already 1500+ indices with above setup.

## Above three nodes are considering as Master & Data nodes. and the cluster configuration details are given below. Index level configurations:

```
"cluster_name" : "aws-es-production",
"index_number_of_shards" : "5",
"index_number_of_replicas" : "0",
"index_translog_flush_threshold_ops" : "100000",
"index_translog_flush_threshold_size" : "1gb",
"index_refresh_interval" : "30s",
"indices_memory_index_buffer_size" : "50%",

```

## Elasticsearch YML configurations: Node1, similar configuration on Node 2 & 3

cluster.name: "aws-es-production"  
node.name: node-IP1  
network.host: IP1  
script.engine.groovy.inline.aggs: on  
script.engine.groovy.inline.search: on  
script.engine.groovy.inline.update: on  
path.data: /EBS1/elasticsearch,/EBS2/elasticsearch,/EBS3/elasticsearch,/EBS4/elasticsearch  
path.logs: /EBS1/elasticsearch/logs  
gateway.expected\_nodes: 3  
gateway.recover\_after\_time: 1m  
gateway.recover\_after\_nodes: 2  
node.master: true  
node.data: true  
discovery.zen.ping.unicast.hosts: ["IP2", "IP3"]  
indices.fielddata.cache.size: 50%  
indices.breaker.fielddata.limit: 60%  
thread\_pool.bulk.queue\_size: 1000  
bootstrap.system\_call\_filter: true  
cluster.info.update.interval: 1m  
cluster.routing.allocation.disk.threshold\_enabled : false

Node 2:  
node.name: node-IP2  
network.host: IP2  
discovery.zen.ping.unicast.hosts: ["IP1", "IP3"]

Node 3:  
node.name: node-IP3  
network.host: IP3  
discovery.zen.ping.unicast.hosts: ["IP1", "IP2"]

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 21, 2019, 7:20am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988/2 "2019-01-21T07:20:55Z")

</div>

That is far too many shards for a cluster that size. I would recommend you read [this blog post](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) and watch [this webinar](https://www.elastic.co/webinars/optimizing-storage-efficiency-in-elasticsearch). Given your quite large bulk queue size, which will just add heap pressure, you might also want to read [this blog post around bulk rejections.](https://www.elastic.co/blog/why-am-i-seeing-bulk-rejections-in-my-elasticsearch-cluster).

I also see that you have groovy scripting enabled. Which version of Elasticsearch are you on?

---

<div class="post-metadata">

**Author:** ![kishore419](https://avatars.discourse-cdn.com/v4/letter/k/b4bc9f/32.png) [@kishore419](https://discuss.elastic.co/u/kishore419)\
**Post date:** [January 21, 2019, 8:41am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988/3 "2019-01-21T08:41:57Z")

</div>

Thank you Christian for quick response.  
i am using Ealsticsearch 5.5.2

---

<div class="post-metadata">

**Author:** ![kishore419](https://avatars.discourse-cdn.com/v4/letter/k/b4bc9f/32.png) [@kishore419](https://discuss.elastic.co/u/kishore419)\
**Post date:** [January 22, 2019, 5:54am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988/4 "2019-01-22T05:54:12Z")

</div>

i read the suggested blogs and saw the webinar sessions.  
here are the challenges i have.  
My cluster is having daily indices, nearly 70+ indices per day.  
\<10 indices size - 30GB to 50GB, remaining indices size is \<1GB  
i could not change my elasticsearch feeder i.e. spark to create the new index when index size matches with suggested value i.e. b/w 20GB to 40GB.

hence i would like to go Forcemerging option. to merge the available indices and save the heapspace/segmentscount/ increase the performance.

i got the following command in elasticsearch website.  
POST /index1,indexes/\_forcemerge?max\_num\_segments=10&flush=true

executed the above command for multiple indices but i could not see any difference in my cluster.  
segments count is not descresed after merging.  
heapmemory is not freed, etc  
GET /\_nodes/stats  
GET /\_cat/segments/index1

Concerns:  
how do i check total associated segments for shard?  
can you please check the above approach and suggest me if any corrections required.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [January 22, 2019, 7:30am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988/5 "2019-01-22T07:30:01Z")

</div>

70 indices per day sound far too much for a small cluster like that, so I would recommend that you change it. If you can not consolidate indices I would recommend that you start using the [rollover index API](https://www.elastic.co/blog/managing-time-based-indices-efficiently). This gives the processes indexing into Elasticsearch an alias to write to that does not change and you can the use [Curator](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html) or the upcoming [Index Lifecycle Management](https://www.elastic.co/guide/en/elasticsearch/reference/6.6/index-lifecycle-management.html) feature to rollover when the indices reach a specific size rather than at a specific time.

You can also look at reducing the number of primary shards to 1 and use the [shrink index API](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/indices-shrink-index.html) to reduce the shard count for existing indices, although given the immense number of shards in your cluster I think more drastic action is required as I described above.

---

<div class="post-metadata">

**Author:** ![kishore419](https://avatars.discourse-cdn.com/v4/letter/k/b4bc9f/32.png) [@kishore419](https://discuss.elastic.co/u/kishore419)\
**Post date:** [January 23, 2019, 12:21pm UTC](https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988/6 "2019-01-23T12:21:20Z")

</div>

Thanks for your inputs.  
i used to follow 'shrink index API' but i saw the below exceptions, since my 5 shards are not in single node and have in 3 nodes.

"illegal\_state\_exception  
index must have all shards allocated on the same node to shrink index"

But i would like to reduce the shards 5 to 1 for 60+ indices.  
how can i resolve the above concern. can you please help if possible.  
thanks in advance.

---

<div class="post-metadata">

**Author:** ![juka](https://avatars.discourse-cdn.com/v4/letter/j/8491ac/32.png) [@juka](https://discuss.elastic.co/u/juka)\
**Post date:** [January 31, 2019, 9:47am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988/7 "2019-01-31T09:47:23Z")

</div>

Hi,  
as already mentioned above use the rollover API for all new indices having a shard size of about 20-30 GB per shard.

Cronjob for it if you have 5 shards and each shard shall be 25GB in Size  
\*/15 \* \* \* \* curl -XPOST '[http://localhost:9200/yourindexaliasname/\_rollover](http://localhost:9200/yourindexaliasname/_rollover)' -d '{ "conditions" : { "max\_size": "125gb" }}' -k

Then reindex the old data using the rollover API and also create alias names for them if access is necessary at all times. You can read more here [https://www.elastic.co/blog/changing-mapping-with-zero-downtime](https://www.elastic.co/blog/changing-mapping-with-zero-downtime)

Here is also some information on reindexing. [https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-reindex.html](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-reindex.html)

BTW. Amazon has some recommendations for Elasticsearch here [https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/sizing-domains.html](https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/sizing-domains.html)

Good Luck

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 28, 2019, 9:47am UTC](https://discuss.elastic.co/t/elasticsearch-cluster-scaling-from-3-nodes-to-5-nodes-to-open-many-indices-shards-best-practices/164988/8 "2019-02-28T09:47:24Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.