# Index lifecyle policy doesn't rollover the index automatically
**URL:** https://discuss.elastic.co/t/index-lifecyle-policy-doesnt-rollover-the-index-automatically/214511
**Category:** Logstash
**Created:** [January 10, 2020, 1:15am UTC](https://discuss.elastic.co/t/index-lifecyle-policy-doesnt-rollover-the-index-automatically/214511 "2020-01-10T01:15:07Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)
#### Post date: [January 10, 2020, 1:15am UTC](https://discuss.elastic.co/t/index-lifecyle-policy-doesnt-rollover-the-index-automatically/214511/1 "2020-01-10T01:15:07Z")
**Objective:**
Write a Logstash configuration that receives logs and sends it to an index using an ILM policy, and rolls over an index after a certain threshold (max 10 documents).
Below is my Logstash configuration for the output plugin.
**Logstash:**
```
output {
elasticsearch {
hosts => ["${HOST}"]
user => "${USERNAME}"
password => "${PASSWORD}"
index => "cloudwatch-testing"
template_name => "cloudwatch"
ilm_rollover_alias => "cloudwatch-testing-alias"
ilm_pattern => "000001"
ilm_policy => "cloudwatch-policy"
id => "cloudwatch"
}
stdout { codec => rubydebug }
}
```
At first, I assumed this will do the work for me but then I realized that if `cloudwatch-testing-000001` doesn't exist, then it will write the logs to `cloudwatch-testing-alias` instead. Therefore, I found these Elasticsearch queries that can create a template and then roll over an index such as cloudwatch-testing-000001 to cloudwatch-testing-000002.
**Problem:** This process is manual. I need an automated solution where I can just send my logs, and after every 10 logs, the index is rolled over and a new index is created. What suggestions / feedback do you have in mind?
**Elasticsearch queries:**
1- Create a template using an existing policy
```
PUT _template/cloudwatch
{
"index_patterns": ["cloudwatch-testing*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "cloudwatch-policy",
"index.lifecycle.rollover_alias": "cloudwatch-testing-alias"
}
}
```
2- Create a new index for logstash to write to, if not it will write to cloudwatch-testing-alias instead which is not the index I would like to write to.
```
PUT cloudwatch-testing-000001
{
"aliases": {
"cloudwatch-testing-alias":{
"is_write_index": true,
"rolled_over" : true
}
}
}
```
3- The below query command rolls the index over manually, but I would like to do this step automatically.
```
POST /cloudwatch-testing-alias/_rollover/cloudwatch-testing-000002
{
"conditions": {
"max_docs": 10
},
"settings": {
"index.number_of_shards": 1
}
}
```
**Policy Attributes:**
`GET /_ilm/policy/cloudwatch-policy`
Returns:
```
{
"cloudwatch-policy" : {
"version" : 3,
"modified_date" : "2020-01-09T23:29:44.942Z",
"policy" : {
"phases" : {
"warm" : {
"min_age" : "30d",
"actions" : {
"set_priority" : {
"priority" : 50
}
}
},
"cold" : {
"min_age" : "30d",
"actions" : {
"freeze" : { },
"set_priority" : {
"priority" : 0
}
}
},
"hot" : {
"min_age" : "0ms",
"actions" : {
"rollover" : {
"max_size" : "50gb",
"max_age" : "30d",
"max_docs" : 10
},
"set_priority" : {
"priority" : 100
}
}
},
"delete" : {
"min_age" : "180d",
"actions" : {
"delete" : { }
}
}
}
}
}
}
```
---
### Author:  [@EZprogramming](https://discuss.elastic.co/u/EZprogramming)
#### Post date: [January 10, 2020, 9:08pm UTC](https://discuss.elastic.co/t/index-lifecyle-policy-doesnt-rollover-the-index-automatically/214511/2 "2020-01-10T21:08:47Z")
Logstash.conf:
```auto
output {
elasticsearch {
hosts => ["${HOST}"]
user => "${USERNAME}"
password => "${PASSWORD}"
index => "logstash-alias"
template_name => "cloudwatch"
ilm_pattern => "000001"
}
stdout { codec => rubydebug }
}
```
and
Template:
```
PUT _template/cloudwatch
{
"index_patterns": ["logstash*"],
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"index.lifecycle.name": "cloudwatch-policy",
"index.lifecycle.rollover_alias": "logstash-alias"
}
}
```
I created cloudwatch-policy from Kibana ILM instead of the Dev Console in Kibana, and then pointed the alias to "logstash-alias". Logstash writes data to the logstash-alias as the index, which then the alias writes the data to the correct indices and rolls them over.
Currently:
green open **logstash-000001** UXl5vscWR2iqk\_PqvEH\_rA 1 1 **31** 0 100.5kb 50.2kb
green open **logstash-000002** 2sMcEzd8QQOtxZKAOUsRIA 1 1 **49** 0 81kb 45.6kb
green open **logstash-000003** rPN-iasORiurCLUgfPqz4A 1 1 **0** 0 460b 230b
The only problem I have noticed is that the alias and policy don't rollover the indices immediately. It takes a while until they do so. If you have a look at the above indices, you can see that they logstash-000001 had 31 documents until it rolled over to logstash-000002, and logstash-000003 got created about 5-10 minutes later. By any chance, anyone knows how to speed up the index rollover?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 7, 2020, 9:08pm UTC](https://discuss.elastic.co/t/index-lifecyle-policy-doesnt-rollover-the-index-automatically/214511/3 "2020-02-07T21:08:49Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.