How to use rolllover index in filebeat configuration

[Rohit_Vyavahare]

I am using filebeat to put all logs from kubernetes into elasticsearch. Here is snippet of my filebeat config file :

    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      index: "%{[kubernetes.namespace]:filebeat}-${PLATFORM}-${PLATFORM_TYPE}-${PLATFORM_REGION}-%{+yyyy.MM.dd}"
      compression_level: "5"
    setup.template:
      name: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}"
      pattern: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-*"

Here it creates index every day.
I want to use rollover index when it reaches one of the conditions from max_age, max_doc_count, mac_size. Is it possible using filebeat kubernetes configuration.

[ruflin]

To make use of rollover you need to trigger each time you want to check for rollovers on the Elasticsearch side. There is currently no way of doing that from the Beats side. What I recommend you is to have a cron job or something similar that triggers the call frequently. Like this you can send your data to rollover indices and make use of it.

[axrayn]

We're currently using index aliases to help with this but we've also moved away from date-based indexes.

E.g. Our filebeat config (via Logstash) goes to the index logstash_filebeat and on the ES side we dynamically manage the index alias and whatever real index its pointing to, say logstash_filebeat-1024.

Just means we don't have to touch our beats config and can roll our indexes as required.

[Rohit_Vyavahare]

Thank you for a reply.
Can curator work here? I saw example: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_rollover.html.
Will it monitor all daily indexes and call rollover on that when one of the conditions meet?

[ruflin]

I haven't tried curator here and only know what I read the docs there. Based on my understanding this looks very promising, so I think the answer is yes.

[theuntergeek (Aaron Mildenstein)]

Yes, it does work that way. Of particular interest for rollover use case are:

1. The pattern option for the count filter. It allows you to keep the last n indices for each matching pattern.
2. The intersect option for the period filter. It allows you to match only indices that have both the minimum and maximum date value within the specified period, rather than one or the other.
3. period_type: absolute allows you to specify absolute date ranges, like all of January 2018, or December 2017 through February 2018.

These filters make it possible to keep only the indices you want to keep much more easily.