Pruning Indexes with 0 documents

Elastic Stack Elasticsearch
1

Hello,

I am looking for the best method to prune indexes with 0 documents.

After we migrated to ILM, we have a few indexes which rollover with 0 documents and this can cause an in-balance across a large cluster where several nodes end up with many shards with 0 documents but not much space utilized as other nodes end up with fewer shards and much more disk utilization. I can't find if we can make an index in delete phase skip min_age if it has 0 documents.

Thanks!

2

Why do you end up with empty indices when using rollover? Are you creating too many small specialized indices? Have you set the max age too low?

3

Hi Christian,

We have several indexes that have 0 docs so far as we have several "Retention" periods for filebeat logging data. Some of those are unused at the moment. We roll over indexes when the shards are approximately 50GiB in size or typically 24 hours old. This is to mainly match previous daily index rollover behavior of Elasticsearch.

Anyhow it looks like I found a potential solution being that I can run curator to prune empty indexes that are an age > the rollover period so we don't accidentally prune the hot index.

1 Like
4

I would just remove the time based rollover and keep the size one in that case.

Curator won't run if it detects ILM though (by default).

5

That effectively makes these no long time-series data, though it would keep the number of 0 doc indexes down to the minimum. It would also permit some time-series data to live much longer than it's desired retention period.

Yup we can just enable it to delete those though.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.