Index houskeeping (ILM)

Elastic Stack Elasticsearch
1

Hi,
I have this question on my mind for a long time. I have many indexes on a Elastic cluster, this cluster is used for application logging so I have plenty of indices separated by application because of they use case. The thing is that all of my indices are daily based (managed by mostly the logstash pipeline or fluentd) also because we need to keep those logs for x days before delete them via ILM. So I ended up with small indexes and plently of used shard. Its there another way to address this, I need to keep logs for lets say lasts 30 days, but I dont want to use a lot of shards

Its there a way to keep 30 days of logs without using 30 indexes? Keep in mind that on the 31 day I just need to delete the oldest data not the whole thing.

If someone has the same issue or question please let me know how can I address this problem

Thanks ind advance,

2

I guess that depends on how you are splitting things into the indices?

3

Thanks for your response but I need a best practice here. I need to keep some data for 30 days, but I don't think that splitting a index daily based its a good approach, because I might be oversharding.

4

Best practise it to use ILM.

But if you're worried about over sharding, then sharing how you are logically splitting data into these indices would be helpful context.
The output from the _cluster/stats?pretty&human API would also be useful.

5

Ok, but is there another solution or approach to keep last 30 days of data, without having 30 indexes and use ILM to delete the 31 day?

6

As you delete complete indices, the time period each index covers determines how much data you delete each time. You can make this a bit more flexible by using rollover as this would allow you to target a specific shard size while allowing indices filled during slower periods to cover longer time periods than a day. Having 30 indices for 30 days retention should however not be a problem, especially if you are on a recent version.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.