Rollover daily indexes

Elastic Stack Elasticsearch
1

Hi.

I am currently looking into the rollover feature in elasticsearch.
At the moment i have timebased indexes from filebeats, eg: filebeat.2017-25-05

I would actually like to still keep filebeat create a new daily index and the the rollover to split these when they reach a certain amount of documents.

But cant figure out from the documentation if this is possible?

Thanks in advance
Peter

2

If I understand you correctly, you want a daily alias, with rolling indices underneath? Like this?

daily-2017.04.23-1
    daily-2017.04.23-000001
    daily-2017.04.23-000002
    daily-2017.04.23-000003
    ....
daily-2017.04.24-1
    daily-2017.04.24-000001
    daily-2017.04.24-000002
    daily-2017.04.24-000003
    ....
daily-2017.04.25-1
    daily-2017.04.25-000001
    daily-2017.04.25-000002
    daily-2017.04.25-000003
    ....

Something like this? While it is possible, it is not automatic. You would have to use some process to make the daily indices/aliases before the rollover time came, so that Beats could just point to it without having to think about it, otherwise Beats would create a new index named daily-YYYY.MM.DD-1 instead of pointing at a rollover-capable alias.

3

@theuntergeek
You got the structure just right.
That answers the question, i dont know i am that confident on relaying on a schedulet job :slight_smile:

Thank you

4

You can set indices to roll over based on document count and/or time, so you could set it up with a pure rolling index and still make sure no index contains data for more than one day if that is the granularity you need in order to manage retention.

5

@Christian_Dahlqvist
That could be an option.
Just have to make sure Curator dont nuke that index.

Nice idea.

6

@Christian_Dahlqvist

How would you control the index-name from logstash when it does a rollover each day and each time it reaches a document count?

/Peter

7

The rollover mechanism controls the underlying index names and Logstash will be indexing into a single write alias. Look at this blog post and the documentation for more details.