How many documents can one Index/shard hold?

Elastic Stack Elasticsearch
1

Hi Elasticsearch Experts,

Do we have any thumb rule that these many documents one index/shard can hold. Let me share my scenario . I have 8-10GB data volume per day in elasticsearch now rather creating per day index I am creating per week index with 5 shards. So my per index(for whole week ) holds around 60-70 GB without replica. Users reporting slowness in the dashboard . My hardware is 16 core 64 GB ram machine . Right now I have assigned 16GGB RAM to ES.

So my Concern are

  1. How would I index data , per day or weekly with 5 shards , or do I go with weekly with 10 shards ?
  2. Do I need to increase my ES RAM like 16 to 20 GB?
  3. Right now I have 2 nodes in cluster , Do I need more nodes to speed up the process ?
  4. Do we have any recommendation from elasticsearch that I should hold this much of data in one shard or Index.

Please provide your expert advice .

Regards
VG

2

A shard is limited to 2 billion docs as that is a lucene limit.

Are you using the Monitoring functionality?

3

Thanks Mark,

For monitoring I have enabled slow query parameter in kibana.With this I am checking which query is taking more than 5 sec and so on.

Regards
Vg

4

Then I'd suggest you use Monitoring to see what's going on/

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.