How many Shards / Replicas

Assuming 270 million documents is what you need to index per day, on the average, how many documents do you have per index? 270mil/15 -> roughly 18 million documents???

By default, ES is configured to create a new index with 5 shards and 1 replica and it seems that's what you have. And you said logstash creates daily indices where each index has 5 shards and 1 replica. This is too much for the amount of data you are indexing per day because each shard can hold up to 2B documents.

I suggest you do the math, adjust logstash to probably create monthly indices, adjust the number of shards and replicas per index before doing a performance test.

1 Like