I am developing elastic search with about 17B documents per index(daily index) and 90 indices in total. Each document will have a size of less than 1Kb.
Since we are using per day index, only one index will be hot write index. And this system is not a heavy read system.
I am planning to go with 30GB per shard and assuming average index size be 10TB, require about 334 shards per index.
What are the general recommendation on building this scale search system on type of instances and other things I need to take care. Should I go with 8 core machines or some 16+ core machines? Hoping different instance type for hot and warm nodes.