Index Retention by Filesize

Elastic Stack Elasticsearch
1

Hi Community,

we are currently building up a logging infrastructure for our network stuff. The plan is like with switching or firewall logs, to let the logs rotate, as soon as the configured disk space is full. I had a look at the documentation and searched a lot in the forum here and with 3rd party forums - I haven't found a solution to my question, so I'm asking here:
We have no unlimited storage under our VMs and it's hard to predict how much traffic will be generated during a specified amount of time. If we fine tune the policy, let's say for a normal business day 100GB will be generated and we have 1,4TB overall storage, it would match 14 days of coverage in theorie. But if there is more network load, so more syslog, more netflow data, we won't make it to 14 days. Had it already, that /var partition maxed out, crashed and elasticsearch wasn't able to start, so I wasn't even able to delete index files via API. Hed to reset the whole config as I was forced to free up space by deleting stuff from /var/lib manually, breaking the database.

So how can we configure, that index files are getting deleted if the sum of files exceds 1,2 TB or let's say I want to keep only 100 index files with 10GB each, so they rotate/delete old files?

Please guide me to the right docs in case I missed anything

Thanks
Chacko42

2

Welcome to our community! :smiley:

Currently there's no way to do that in the stack, retention is done by age only. I would strongly suggest you raise a feature request on GitHub to have this considered. I can see ILM index rollover and delete based on index size · Issue #93289 · elastic/elasticsearch · GitHub which is similar but not exact.

1 Like
3

I believe Curator unlike ILM might have the ability to delete the oldest indices when specified total storage has been exceeded, so I would recommend checking it out.

1 Like
4

Thanks for having me :wink:

Okay, I'm a little bit surprised, thought that's a default requirement, but I guess the rest got more storage ressource than I am. Then I will go for an API script, which will be triggered by chron regularly and delete the indexes-over-threshold. I will have a look at the feature requests
Many thanks!

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.