Recommended shard size and data node instance type in AWS

Elastic Stack Elasticsearch
1

I am developing elastic search with about 17B documents per index(daily index) and 90 indices in total. Each document will have a size of less than 1Kb.

Since we are using per day index, only one index will be hot write index. And this system is not a heavy read system.

I am planning to go with 30GB per shard and assuming average index size be 10TB, require about 334 shards per index.

What are the general recommendation on building this scale search system on type of instances and other things I need to take care. Should I go with 8 core machines or some 16+ core machines? Hoping different instance type for hot and warm nodes.

2

May be you should use hourly indices?

May I suggest you look at the following resources about sizing:

https://www.elastic.co/elasticon/conf/2016/sf/quantitative-cluster-sizing

And https://www.elastic.co/webinars/using-rally-to-get-your-elasticsearch-cluster-size-right

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.