Max Shards allowed per Index

Elastic Stack Elasticsearch
1

We are planning to have 1 index which will host ~150TB of data. Ideally it is suggested to have 1 shard per 50GB. So i may end up in 3000 Shards!!!

does ES really works with 3000 shards on 1 index? We don't have problem with resources like cores,memory and disk...We can scale any number of Elastic search nodes.

All we concerned is...Does ES supports at this scale? any standard guidelines for number of shards allowed per index?

Please help us on this for better decision making of our product.

2

Can you explain your use-case a bit more? What would be the reason of putting everything into a single index? How are your queries structured?

In general, the 50GB per shard rule is not a strict rule, but a recommendation. Before having thousands of shards, it makes sense to bend this rule.

In general a shard is just a lucene index. It does not matter if you query 3000 shards across 3000 indices or 3000 shards across a single index, the same work would need to be done in the background. That's one of the reasons why it makes sense to keep the number of shards low.

Hope that helps as a start.

3

Basically we have 1 index where customer keeps uploading their avg 1MB doc. We are expecting almost 150TB data in 2 years. Some how we cannot distribute that to too many indices and has to keep only 1 index. And we keep this data forever until customer deletes their files.

Actually we just tried a run keeping 1 index with 1800 shards. ingested ~6TB of data. Now we see that we cannot go more than 15searches/sec. We have 6 data nodes each of 8 cores. With just 15searches/sec, we are seeing search threads are hitting max. as per elastic search stats, query rate is touching 30K/sec which is very huge for just 15 searches/sec.

That is where we are trying to find what is optimal shards we can keep.

4

How about having a single index with a few more shards, but having all the data for one user in a single shard?

See this chapter in the definitive guide.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.