Rollover daily indexes

Hi.

I am currently looking into the rollover feature in elasticsearch.
At the moment i have timebased indexes from filebeats, eg: filebeat.2017-25-05

I would actually like to still keep filebeat create a new daily index and the the rollover to split these when they reach a certain amount of documents.

But cant figure out from the documentation if this is possible?

Thanks in advance
Peter

If I understand you correctly, you want a daily alias, with rolling indices underneath? Like this?

daily-2017.04.23-1
    daily-2017.04.23-000001
    daily-2017.04.23-000002
    daily-2017.04.23-000003
    ....
daily-2017.04.24-1
    daily-2017.04.24-000001
    daily-2017.04.24-000002
    daily-2017.04.24-000003
    ....
daily-2017.04.25-1
    daily-2017.04.25-000001
    daily-2017.04.25-000002
    daily-2017.04.25-000003
    ....

Something like this? While it is possible, it is not automatic. You would have to use some process to make the daily indices/aliases before the rollover time came, so that Beats could just point to it without having to think about it, otherwise Beats would create a new index named daily-YYYY.MM.DD-1 instead of pointing at a rollover-capable alias.

@theuntergeek
You got the structure just right.
That answers the question, i dont know i am that confident on relaying on a schedulet job :slight_smile:

Thank you

You can set indices to roll over based on document count and/or time, so you could set it up with a pure rolling index and still make sure no index contains data for more than one day if that is the granularity you need in order to manage retention.

@Christian_Dahlqvist
That could be an option.
Just have to make sure Curator dont nuke that index.

Nice idea.

@Christian_Dahlqvist

How would you control the index-name from logstash when it does a rollover each day and each time it reaches a document count?

/Peter

The rollover mechanism controls the underlying index names and Logstash will be indexing into a single write alias. Look at this blog post and the documentation for more details.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.