Hi Elasticsearch Experts,
Do we have any thumb rule that these many documents one index/shard can hold. Let me share my scenario . I have 8-10GB data volume per day in elasticsearch now rather creating per day index I am creating per week index with 5 shards. So my per index(for whole week ) holds around 60-70 GB without replica. Users reporting slowness in the dashboard . My hardware is 16 core 64 GB ram machine . Right now I have assigned 16GGB RAM to ES.
So my Concern are
- How would I index data , per day or weekly with 5 shards , or do I go with weekly with 10 shards ?
- Do I need to increase my ES RAM like 16 to 20 GB?
- Right now I have 2 nodes in cluster , Do I need more nodes to speed up the process ?
- Do we have any recommendation from elasticsearch that I should hold this much of data in one shard or Index.
Please provide your expert advice .