# Rollover Indeces using Logstash and Elastic **URL:** \ **Category:** Logstash\ **Tags:** ilm-index-lifecycle-management\ **Created:** [July 31, 2020, 2:23pm UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362 "2020-07-31T14:23:16Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![Kosodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosodrom/32/99278_2.png) [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Post date:** [July 31, 2020, 2:23pm UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/1 "2020-07-31T14:23:16Z")
Hi guys, I have following use case: I want to create new index every day, therefore my logstash output configuraiton looks as this: ``` output { elasticsearch { index => "metricbeat-linux-vms-%{+YYYY.MM.dd}" ``` I also have an ILM policy which defines rollover after 50gb and I have an index template which uses this ilm policy and is applied to all indeces with `metricbeat-linux-vms*`. Where do I define that after 50gbs are reached a new index is created like: `metricbeat-linux-vms-2020-07-31-000001`? Thank you very much ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 3, 2020, 10:43pm UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/2 "2020-08-03T22:43:02Z")
We aren't all guys 🙂 What does your ILM policy look like? ---
**Author:** ![Kosodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosodrom/32/99278_2.png) [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Post date:** [August 4, 2020, 7:07am UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/3 "2020-08-04T07:07:04Z")
Thank you for your response. Sorry, by saying "guys" I do not have specific gender in my mind. So my ILM policy is pretty easy: I want to create a new index onces my existing one has reached 50GB. I created in in Kibana. And I have an Indx template, which is using this policy: ``` { "index": { "lifecycle": { "name": "ilm-metricbeat-my-policy" } ``` But my struggle is: I have daily indeces. Every day I create a new index by using this configuration in logstash output: `index => "metricbeat-linux-vms-%{+YYYY.MM.dd}"` What I don't understand is: How can I define the creation of new index after 50 GB like `"metricbeat-linux-vms-%{+YYYY.MM.dd}-000001"` So during the day I have following indeces: `"metricbeat-linux-vms-2020.08.04"` `"metricbeat-linux-vms-2020.08.04-000001"` `"metricbeat-linux-vms-2020.08.04-000002"` next day: `"metricbeat-linux-vms-2020.08.05"` ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 9, 2020, 11:20pm UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/4 "2020-08-09T23:20:15Z")
> [@Kosodrom](#): > > So during the day I have following indeces: > > `"metricbeat-linux-vms-2020.08.04"` > `"metricbeat-linux-vms-2020.08.04-000001"` > `"metricbeat-linux-vms-2020.08.04-000002"` > > next day: > > `"metricbeat-linux-vms-2020.08.05"` That's not now ILM is designed. The timestamp in the index name is when the policy was first used. ---
**Author:** ![Kosodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosodrom/32/99278_2.png) [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Post date:** [August 10, 2020, 6:53am UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/5 "2020-08-10T06:53:59Z")
So how would you do a rollover on daily indeces ? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 10, 2020, 6:58am UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/6 "2020-08-10T06:58:18Z")
You define it in the policy as `max_age`. ---
**Author:** ![Kosodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosodrom/32/99278_2.png) [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Post date:** [August 12, 2020, 8:19am UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/7 "2020-08-12T08:19:29Z")
What would be the rollover alias in this case? I need to provide it in the index template and I am not sure what I need to provide here. So you would suggest to leave out the timestamp from the index name and to have instead an index like: `metricbeat-linux-vms` ? I have problems to understand how I can create daily indeces then. My idea was to create an index every day to maintain them easier (for example delete all index with `metricbeat-linux-vms-2020.08*`" ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 12, 2020, 8:43am UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/8 "2020-08-12T08:43:23Z")
The idea is that ILM knows how old the indices are, even if they aren't created with a date in the name. So all you need to do there is set the retention period in the policy. ---
**Author:** ![Kosodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosodrom/32/99278_2.png) [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Post date:** [August 12, 2020, 9:58am UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/9 "2020-08-12T09:58:53Z")
Ok. Thanks. Could you please explain what the desired approach would be like using filebeat -\> logstash -\> elasticsearch for that? As far as I understood: In filebeat I have: ``` output.logstash: hosts: ["logstash.host:5044"] ``` In logstash I have: ``` output { elasticsearch { hosts => ["https://elastic.host:9200"] index => "metricbeat-linux-vms" } } ``` I have a policy `ilm-metricbeat`with `max_age: 1d` and enabled rolle over I have an index template: ``` { "index": { "lifecycle": { "name": "ilm-metricbeat", "rollover_alias": "metricbeat-linux-vms" } } ``` This configuration force the creation of new index after 1d has passed. Am I correct so far? Things I do not understand yet: 1. How can I distiguishe the index created yesterday from the one created today, since they have the same name 2. How I can access the logs from yesterday, when a new index was created today? 3. How can I use variables in the index name, when I decide later to seperate my logs or metrics into multiple indeces? 4. How can I acess the indeces created, let's say during the last month to backup them? Thank you ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 17, 2020, 11:39pm UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/10 "2020-08-17T23:39:18Z")
> [@Kosodrom](#): > > This configuration force the creation of new index after 1d has passed. Am I correct so far? If that is what you have in the policy under `max_age`, then yes. 1. The numeric prefix will change, starting from `000001 `. So each day will increment by +1 2. Access them how? 3. You split them out and use different ILM policies for them 4. Use [SLM](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshot-lifecycle-management.html) ---
**Author:** ![Kosodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosodrom/32/99278_2.png) [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Post date:** [August 21, 2020, 10:25am UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/11 "2020-08-21T10:25:13Z")
> The numeric prefix will change, starting from `000001 ` . So each day will increment by +1 Where do you define that? in the logstash pipeline configuration or on the ilm configuration or in the index template? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 22, 2020, 5:31am UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/12 "2020-08-22T05:31:41Z")
It's built in behaviour. ---
**Author:** ![Kosodrom](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kosodrom/32/99278_2.png) [@Kosodrom](https://discuss.elastic.co/u/Kosodrom)\ **Post date:** [August 23, 2020, 4:12pm UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/13 "2020-08-23T16:12:47Z")
What is inside the rollover\_alias of the index template then ? Just 1:1 the name of the index ? So when I want to write into index: `metricbeat-linux-vms` my rollover\_alias will be `metricbeat-linux-vms` and when I set the `max_age` to 1 day after a day has passed I will see an index called: `metricbeat-linux-vms` and one called ` metricbeat-linux-vms-00001` Is it correct? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 23, 2020, 10:52pm UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/14 "2020-08-23T22:52:14Z")
> [@Kosodrom](#): > > `metricbeat-linux-vms` That is the alias, so it's not an index. > [@Kosodrom](#): > > `metricbeat-linux-vms-00001` That would be the very first index that is created that is attached to the above alias. The next day's index would be `metricbeat-linux-vms-00002`, and so on. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 20, 2020, 10:52pm UTC](https://discuss.elastic.co/t/rollover-indeces-using-logstash-and-elastic/243362/15 "2020-09-20T22:52:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.