Most right architecture for my cluster

Elastic Stack Elasticsearch
1

Hey all,
Currently I have a 3 nodes cluster with the following spec:
Every server has 12 cores, 32gb of memory and 3 TB of NVME disks.

I have total of 6.5TB of data, spreaded on 30 daily indices and another four indices with 10-200 million of documents.

Every day I bulk between 200-350GB of data.
Every daily index have 3 primary shards with no replicas.

Im having heap issues due to the amount of indexing we are doing, and overall slowness, therefore I order another 2 servers, with the same spec.

Now, I have some options regarding the amount of master nodes:

  1. 5 nodes that are both master and data nodes
  2. 3 master+data nodes and 2 data nodes

My main goal is to have a faster cluster (bulks are taking 1-15 seconds per bulk) and to have a stable cluster (my cluster is getting freezed every week due to OOM)

Thanks!

2

I think that you probably need a bit more shards for the current index if you want to speed up indexing.
Then the next day you might want to use shrink API to reduce that number.

May be you should add more nodes and think of a hot-warm design?

I'm dropping here some documentation which might be helpful.

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right

3

This webinar might also be useful as you are suffering from heap pressure. A good way to consistently get reasonably sized shards is to use the rollover API. This allows to to create indices when needed based on age and/or size instead of doing this rigidly on a daily basis, and is supported by the new Index Lifecycle Management feature available in version 6.6. This will allow you to use the same number of primary shards as you have nodes, but cut indices more or less frequently than now depending on indexing volumes.

4

Go for Option Two
3 master + 2 data nodes
You need backup so go for 1 replica. If a node fails your cluster will still function.
As mentioned above use the rollover API and create shard sizes between 20-30 GB per shard:
*/15 * * * * curl -XPOST 'http://localhost:9200/yourindexaliasname/_rollover' -d '{ "conditions" : { "max_size": "125gb" }}' -k
If you have 5 shards and each shard shall be 25GB in Size

If you can't shrink the old data reindex it and get rid of too many indices!

Nice to read and get a better understanding on scaling your cluster.
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/sizing-domains.html
https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/petabyte-scale.html
--- RAM Storage Ratio ---
https://www.elastic.co/guide/en/cloud-enterprise/1.1/ece-manage-node-configurations.html

Good luck!

5

Given that you have a good amount of data coming in per day I think you might be better off right-sizing your shards from the outset and not have to go through using the shrink index API. Based on your heap size and amount of storage per node, aiming for an average shard size between 25GB and 30GB might be a good starting point.

6

Thanks everyone, it was very helpful! I will read all the materials and will test everything.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.