Average shard size

Elastic Stack Elasticsearch
1

Hello learned ladies and gentlemen,

I am trying to get a feel for a good average shard size.

I have a 6 node cluster
3 nodes are master, data, ingest
3 nodes are data
Each node has 64GB memory and 30GB java heap size
There are about 3100 indices
Each index has 5 shards and 1 replica
Total shard size is 25TB
Average shard size is 685 MB
There are about 36000 shards
We mostly, ingest then do kibana searches, dashboards

Does that sound about right?
Does anyone thing I have too many shards?

Thanks in advance
David

2

I would say that is clearly too many shards. Have a look at this blog post for some practical guidance.

3

Yes, number of shards are on the higher end. In fact average data per shard is quite low (685MB). You can easily increase data per shard to somewhere around 5G, and reduce the total number of shards.

4

Thanks guys. Valuable input. Cheers

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.