Rollover daily indexes

(Peter Bøndum) #1


I am currently looking into the rollover feature in elasticsearch.
At the moment i have timebased indexes from filebeats, eg: filebeat.2017-25-05

I would actually like to still keep filebeat create a new daily index and the the rollover to split these when they reach a certain amount of documents.

But cant figure out from the documentation if this is possible?

Thanks in advance

(Aaron Mildenstein) #2

If I understand you correctly, you want a daily alias, with rolling indices underneath? Like this?


Something like this? While it is possible, it is not automatic. You would have to use some process to make the daily indices/aliases before the rollover time came, so that Beats could just point to it without having to think about it, otherwise Beats would create a new index named daily-YYYY.MM.DD-1 instead of pointing at a rollover-capable alias.

(Peter Bøndum) #3

You got the structure just right.
That answers the question, i dont know i am that confident on relaying on a schedulet job :slight_smile:

Thank you

(Christian Dahlqvist) #4

You can set indices to roll over based on document count and/or time, so you could set it up with a pure rolling index and still make sure no index contains data for more than one day if that is the granularity you need in order to manage retention.

(Peter Bøndum) #5

That could be an option.
Just have to make sure Curator dont nuke that index.

Nice idea.

(Peter Bøndum) #6


How would you control the index-name from logstash when it does a rollover each day and each time it reaches a document count?


(Christian Dahlqvist) #7

The rollover mechanism controls the underlying index names and Logstash will be indexing into a single write alias. Look at this blog post and the documentation for more details.

(system) #8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.