# How to use rolllover index in filebeat configuration

**URL:** https://discuss.elastic.co/t/how-to-use-rolllover-index-in-filebeat-configuration/121557
**Category:** Beats
**Tags:** filebeat
**Created:** [February 26, 2018, 7:47pm UTC](https://discuss.elastic.co/t/how-to-use-rolllover-index-in-filebeat-configuration/121557 "2018-02-26T19:47:55Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![Rohit\_Vyavahare](https://avatars.discourse-cdn.com/v4/letter/r/f4b2a3/32.png) [@Rohit\_Vyavahare](https://discuss.elastic.co/u/Rohit_Vyavahare)
#### Post date: [February 26, 2018, 7:47pm UTC](https://discuss.elastic.co/t/how-to-use-rolllover-index-in-filebeat-configuration/121557/1 "2018-02-26T19:47:55Z")

</div>

I am using filebeat to put all logs from kubernetes into elasticsearch. Here is snippet of my filebeat config file :

```
    output.elasticsearch:
      hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
      username: ${ELASTICSEARCH_USERNAME}
      password: ${ELASTICSEARCH_PASSWORD}
      index: "%{[kubernetes.namespace]:filebeat}-${PLATFORM}-${PLATFORM_TYPE}-${PLATFORM_REGION}-%{+yyyy.MM.dd}"
      compression_level: "5"
    setup.template:
      name: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}"
      pattern: "%{[kubernetes.namespace]:filebeat}-%{[beat.version]}-*"

```

Here it creates index every day.  
I want to use rollover index when it reaches one of the conditions from max\_age, max\_doc\_count, mac\_size. Is it possible using filebeat kubernetes configuration.

---

<div class="post-metadata">

### Author: ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [February 27, 2018, 3:59am UTC](https://discuss.elastic.co/t/how-to-use-rolllover-index-in-filebeat-configuration/121557/2 "2018-02-27T03:59:47Z")

</div>

To make use of rollover you need to trigger each time you want to check for rollovers on the Elasticsearch side. There is currently no way of doing that from the Beats side. What I recommend you is to have a cron job or something similar that triggers the call frequently. Like this you can send your data to rollover indices and make use of it.

---

<div class="post-metadata">

### Author: ![axrayn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/axrayn/32/56503_2.png) [@axrayn](https://discuss.elastic.co/u/axrayn)
#### Post date: [February 27, 2018, 7:17am UTC](https://discuss.elastic.co/t/how-to-use-rolllover-index-in-filebeat-configuration/121557/3 "2018-02-27T07:17:38Z")

</div>

We're currently using index aliases to help with this but we've also moved away from date-based indexes.

E.g. Our filebeat config (via Logstash) goes to the index logstash\_filebeat and on the ES side we dynamically manage the index alias and whatever real index its pointing to, say logstash\_filebeat-1024.

Just means we don't have to touch our beats config and can roll our indexes as required.

---

<div class="post-metadata">

### Author: ![Rohit\_Vyavahare](https://avatars.discourse-cdn.com/v4/letter/r/f4b2a3/32.png) [@Rohit\_Vyavahare](https://discuss.elastic.co/u/Rohit_Vyavahare)
#### Post date: [February 27, 2018, 6:23pm UTC](https://discuss.elastic.co/t/how-to-use-rolllover-index-in-filebeat-configuration/121557/4 "2018-02-27T18:23:03Z")

</div>

Thank you for a reply.  
Can curator work here? I saw example: [https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex\_rollover.html](https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_rollover.html).  
Will it monitor all daily indexes and call rollover on that when one of the conditions meet?

---

<div class="post-metadata">

### Author: ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [February 28, 2018, 8:00am UTC](https://discuss.elastic.co/t/how-to-use-rolllover-index-in-filebeat-configuration/121557/5 "2018-02-28T08:00:49Z")

</div>

I haven't tried curator here and only know what I read the docs there. Based on my understanding this looks very promising, so I think the answer is yes.

---

<div class="post-metadata">

### Author: ![theuntergeek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/theuntergeek/32/44961_2.png) [@theuntergeek](https://discuss.elastic.co/u/theuntergeek)
#### Post date: [March 6, 2018, 8:37pm UTC](https://discuss.elastic.co/t/how-to-use-rolllover-index-in-filebeat-configuration/121557/6 "2018-03-06T20:37:08Z")

</div>

Yes, it does work that way. Of particular interest for rollover use case are:

1. The [`pattern`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.4/filtertype_count.html#_pattern_based_sorting) option for the [`count` filter](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.4/filtertype_count.html). It allows you to keep the last _n_ indices for each matching pattern.
2. The [`intersect`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.4/fe_intersect.html) option for the [`period` filter](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.4/filtertype_period.html). It allows you to match only indices that have both the minimum _and_ maximum date value within the specified period, rather than one or the other.
3. [`period_type: absolute`](https://www.elastic.co/guide/en/elasticsearch/client/curator/5.4/filtertype_period.html#_absolute_periods) allows you to specify absolute date ranges, like all of January 2018, or December 2017 through February 2018.

These filters make it possible to keep only the indices you want to keep much more easily.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 3, 2018, 8:37pm UTC](https://discuss.elastic.co/t/how-to-use-rolllover-index-in-filebeat-configuration/121557/7 "2018-04-03T20:37:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
