Max_docs when using ILM and automatic rollover

I'm exploring the automatic rollover for ILM and are looking for some sensible default when to rollover. max_age is straight forward e.g. 1 day, 7 days e.g. 1 month. When it comes to max_size I'll guess it depends. But let's pretend I'd like to shrink my index to 1 shard when it hits the warm phase 20GB could be a sensible default, https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster so we're not ending up with a too large single shard in the index.

When it comes to number of documents I haven't found any real guide lines but this post indicates 3M per shard https://thoughts.t37.net/designing-the-perfect-elasticsearch-cluster-the-almost-definitive-guide-e614eabc1a87#3615 but I'll guess it depends as well. My logs are typically either request logs or structured application logs. Are there any other resources/benchmarks for how to select the number of documents?

Would be interested in some dialogue around this as well. I've found that performance seems to do well at 3M documents per shard, but I'm also seeing that my shard sizes as a result are around 1GB. Seems they can be as large as 30GB or so. And my managed cluster has a ton of available disk usage but I have to be mindful of not exceeding 20 shards per GB heap.

Pro tips here would be much appreciated. Thanks!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.