Elasticsearch cluster scaling from 3 nodes to 5 nodes to open many indices/shards, best practices

Elastic Stack Elasticsearch
1

We are using three node Elasticsearch cluster in AWS Cloud.
here are the configurations.
Instance Type: M4 4X Large (Three nodes), each node configurations are given below.
CPU, 16 cores, RAM 64GB, EBS volumes: 4TB * 4 volumes
EBS Volume Utilizations: 2.5Tb * 4 volumes.
JVM heapsapce : 31.5GB, virtual memory : 10GB

With the above configuration we could able to 'open 520 indices active_primary_shards: 2800' from 6300 indexes (Appx 39000 shards).
JVM utlization details are below:

ip heap.percent ram.percent cpu load_1m load_5m load_15m
IP1 63 99 5 0.60 0.46 0.45mdi
IP2 37 98 4 0.45 0.34 0.32mdi
IP3 56 96 4 0.25 0.19 0.25mdi

Master node: * and node is : node-IP1

Requirement:

we have to open 1750 indices (Appx 11600 shards) in the above cluster to meet my customers requirement.
Unfortunately we don't have sufficient Staging stack to test the following proposals, hence please look the below proposals and suggest me on best approach/possible precautions.

Proposal 1:
We would like to join/add 2 more nodes i.e. M4 4X Large nodes to the above cluster (make 5 node cluster).
-> We will be configured JVM heap space as 31.5GB similar to the other nodes.
-> Two new nodes would be considered as only Master nodes (data node flag - false)
-> No EBS volumes for new nodes, since existing EBS volume size is sufficient for my requirement
-> minimum master nodes 3 and data nodes 2

JVM heap space would be increased with the above proposal and we could open atlready 1500+ indices with above setup.
please share your thoughts

Proposal 2:
We would like to join/add 2 more nodes i.e. M4 4X Large nodes to the above cluster (make 5 node cluster).
-> We will be configured JVM heap space as 31.5GB similar to the other nodes.
-> Two new nodes would be considered as Master and data nodes
-> Only one EBS volume per node and capacity is 10TB.
-> minimum master nodes 3 and data nodes 2

JVM heap space would be increased with the above proposal and we could open already 1500+ indices with above setup.
please share your thoughts

Proposal 3:
Configuration is similar to the existing cluster,
minimum master nodes 3 and data nodes 2

JVM heap space would be increased with the above proposal and we could open already 1500+ indices with above setup.

Above three nodes are considering as Master & Data nodes. and the cluster configuration details are given below.
Index level configurations:

"cluster_name" : "aws-es-production",
"index_number_of_shards" : "5",
"index_number_of_replicas" : "0",
"index_translog_flush_threshold_ops" : "100000",
"index_translog_flush_threshold_size" : "1gb",
"index_refresh_interval" : "30s",
"indices_memory_index_buffer_size" : "50%",

Elasticsearch YML configurations: Node1, similar configuration on Node 2 & 3

cluster.name: "aws-es-production"
node.name: node-IP1
network.host: IP1
script.engine.groovy.inline.aggs: on
script.engine.groovy.inline.search: on
script.engine.groovy.inline.update: on
path.data: /EBS1/elasticsearch,/EBS2/elasticsearch,/EBS3/elasticsearch,/EBS4/elasticsearch
path.logs: /EBS1/elasticsearch/logs
gateway.expected_nodes: 3
gateway.recover_after_time: 1m
gateway.recover_after_nodes: 2
node.master: true
node.data: true
discovery.zen.ping.unicast.hosts: ["IP2", "IP3"]
indices.fielddata.cache.size: 50%
indices.breaker.fielddata.limit: 60%
thread_pool.bulk.queue_size: 1000
bootstrap.system_call_filter: true
cluster.info.update.interval: 1m
cluster.routing.allocation.disk.threshold_enabled : false

Node 2:
node.name: node-IP2
network.host: IP2
discovery.zen.ping.unicast.hosts: ["IP1", "IP3"]

Node 3:
node.name: node-IP3
network.host: IP3
discovery.zen.ping.unicast.hosts: ["IP1", "IP2"]

2

That is far too many shards for a cluster that size. I would recommend you read this blog post and watch this webinar. Given your quite large bulk queue size, which will just add heap pressure, you might also want to read this blog post around bulk rejections..

I also see that you have groovy scripting enabled. Which version of Elasticsearch are you on?

1 Like
3

Thank you Christian for quick response.
i am using Ealsticsearch 5.5.2

4

i read the suggested blogs and saw the webinar sessions.
here are the challenges i have.
My cluster is having daily indices, nearly 70+ indices per day.
<10 indices size - 30GB to 50GB, remaining indices size is <1GB
i could not change my elasticsearch feeder i.e. spark to create the new index when index size matches with suggested value i.e. b/w 20GB to 40GB.

hence i would like to go Forcemerging option. to merge the available indices and save the heapspace/segmentscount/ increase the performance.

i got the following command in elasticsearch website.
POST /index1,indexes/_forcemerge?max_num_segments=10&flush=true

executed the above command for multiple indices but i could not see any difference in my cluster.
segments count is not descresed after merging.
heapmemory is not freed, etc
GET /_nodes/stats
GET /_cat/segments/index1

Concerns:
how do i check total associated segments for shard?
can you please check the above approach and suggest me if any corrections required.

5

70 indices per day sound far too much for a small cluster like that, so I would recommend that you change it. If you can not consolidate indices I would recommend that you start using the rollover index API. This gives the processes indexing into Elasticsearch an alias to write to that does not change and you can the use Curator or the upcoming Index Lifecycle Management feature to rollover when the indices reach a specific size rather than at a specific time.

You can also look at reducing the number of primary shards to 1 and use the shrink index API to reduce the shard count for existing indices, although given the immense number of shards in your cluster I think more drastic action is required as I described above.

6

Thanks for your inputs.
i used to follow 'shrink index API' but i saw the below exceptions, since my 5 shards are not in single node and have in 3 nodes.

"illegal_state_exception
index must have all shards allocated on the same node to shrink index"

But i would like to reduce the shards 5 to 1 for 60+ indices.
how can i resolve the above concern. can you please help if possible.
thanks in advance.

7

Hi,
as already mentioned above use the rollover API for all new indices having a shard size of about 20-30 GB per shard.

Cronjob for it if you have 5 shards and each shard shall be 25GB in Size
*/15 * * * * curl -XPOST 'http://localhost:9200/yourindexaliasname/_rollover' -d '{ "conditions" : { "max_size": "125gb" }}' -k

Then reindex the old data using the rollover API and also create alias names for them if access is necessary at all times. You can read more here https://www.elastic.co/blog/changing-mapping-with-zero-downtime

Here is also some information on reindexing. https://www.elastic.co/guide/en/elasticsearch/reference/2.4/docs-reindex.html

BTW. Amazon has some recommendations for Elasticsearch here https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/sizing-domains.html

Good Luck

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.