# ILM Policy overriden by Logstash

**URL:** <https://discuss.elastic.co/t/ilm-policy-overriden-by-logstash/200031>\
**Category:** Logstash\
**Tags:** ilm-index-lifecycle-management\
**Created:** [September 18, 2019, 2:28pm UTC](https://discuss.elastic.co/t/ilm-policy-overriden-by-logstash/200031 "2019-09-18T14:28:45Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![bharath.krishn2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bharath.krishn2/32/130240_2.png) [@bharath.krishn2](https://discuss.elastic.co/u/bharath.krishn2)\
**Post date:** [September 18, 2019, 2:28pm UTC](https://discuss.elastic.co/t/ilm-policy-overriden-by-logstash/200031/1 "2019-09-18T14:28:45Z")

</div>

I have a requirement for pull the data from two separate sources. I'm using 'file' and 'jdbc' as input plugin in the logstash configuration.  
In the output, I've segragated elasticsearch indices based on the types.  
Below is my output config:

output {

if [type] == "abc\_logs" {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
manage\_template =\> true  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
template =\> "/etc/logstash/conf.d/abc-template.json"  
template\_name =\> "custom-abc-template"  
ilm\_enabled =\> true  
ilm\_pattern =\> "000001"  
}  
}

if [type] == "def\_transactions" {  
elasticsearch {  
hosts =\> ["localhost:9200"]  
manage\_template =\> true  
index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
template =\> "/etc/logstash/conf.d/def-template.json"  
template\_name =\> "custom-def-template"  
ilm\_enabled =\> true  
ilm\_pattern =\> "500001"  
}  
}  
}

Below are my two different index templates:

abc-template.json:

{  
"index\_patterns" : ["abclogstash-0\*"],  
"settings" : {  
"number\_of\_shards" : 5,  
"index.lifecycle.name":"abc-logstash-policy",  
"index.lifecycle.rollover\_alias":"abclogstash"  
},  
"aliases" : {  
"abc-alias" : {}  
}  
}

def-template.json:

{  
"index\_patterns" : ["deflogstash-0\*"],  
"settings" : {  
"number\_of\_shards" : 5,  
"index.lifecycle.name":"def-logstash-policy",  
"index.lifecycle.rollover\_alias":"deflogstash"

```
},
"aliases" : {
    "def-alias" : {}
}

```

}

Now the problem is the when I'm starting the ELK container, the template is being overriden by logstash and creating same index.lifecycle.rollover\_alias for both the indices. This is causing problem in elasticsearch. Below are the logs.

==\> /var/log/elasticsearch/elasticsearch.log \<==  
[2019-09-18T15:15:09,916][INFO][o.e.c.m.MetaDataIndexTemplateService] [3Qz6On0] adding template [custom-abc-template] for index patterns [logstash-_]  
[2019-09-18T15:15:09,948][INFO][o.e.c.m.MetaDataIndexTemplateService] [3Qz6On0] adding template [custom-def-template] for index patterns [logstash-_]  
[2019-09-18T15:15:10,068][INFO][o.e.c.m.MetaDataCreateIndexService] [3Qz6On0] [logstash-000001] creating index, cause [api], templates [custom-abc-template, custom-def-template], shards [5]/[1], mappings [doc]

==\> /var/log/logstash/logstash-plain.log \<==  
[2019-09-18T15:15:09,318][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}}  
[2019-09-18T15:15:09,539][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"}  
[2019-09-18T15:15:09,613][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6}  
[2019-09-18T15:15:09,616][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}  
[2019-09-18T15:15:09,644][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]}  
[2019-09-18T15:15:09,666][INFO][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=\>{:removed=\>, :added=\>[[http://localhost:9200/](http://localhost:9200/)]}}  
[2019-09-18T15:15:09,694][WARN][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=\>"[http://localhost:9200/](http://localhost:9200/)"}  
[2019-09-18T15:15:09,707][INFO][logstash.outputs.elasticsearch] ES Output version determined {:es\_version=\>6}  
[2019-09-18T15:15:09,707][WARN][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document \_type {:es\_version=\>6}  
[2019-09-18T15:15:09,713][INFO][logstash.outputs.elasticsearch] New Elasticsearch output {:class=\>"LogStash::Outputs::ElasticSearch", :hosts=\>["[//localhost:9200](https://localhost:9200)"]}  
[2019-09-18T15:15:09,790][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>"/etc/logstash/conf.d/abc-template.json"}  
[2019-09-18T15:15:09,795][INFO][logstash.outputs.elasticsearch] Using mapping template from {:path=\>"/etc/logstash/conf.d/def-template.json"}  
[2019-09-18T15:15:09,817][INFO][logstash.outputs.elasticsearch] Overwriting index patterns, as ILM is enabled.  
[2019-09-18T15:15:09,824][INFO][logstash.outputs.elasticsearch] Overwriting index lifecycle name and rollover alias as ILM is enabled.  
[2019-09-18T15:15:09,827][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"index\_patterns"=\>"logstash-_", "settings"=\>{"number\_of\_shards"=\>5, "index.lifecycle.name"=\>"logstash-policy", "index.lifecycle.rollover\_alias"=\>"logstash"}, "aliases"=\>{"mse-alias"=\>{}}}}  
[2019-09-18T15:15:09,835][INFO][logstash.outputs.elasticsearch] Overwriting index patterns, as ILM is enabled.  
[2019-09-18T15:15:09,836][INFO][logstash.outputs.elasticsearch] Overwriting index lifecycle name and rollover alias as ILM is enabled.  
[2019-09-18T15:15:09,843][INFO][logstash.outputs.elasticsearch] Attempting to install template {:manage\_template=\>{"index\_patterns"=\>"logstash-_", "settings"=\>{"number\_of\_shards"=\>5, "index.lifecycle.name"=\>"logstash-policy", "index.lifecycle.rollover\_alias"=\>"logstash"}, "aliases"=\>{"atg-alias"=\>{}}}}  
[2019-09-18T15:15:09,860][INFO][logstash.outputs.elasticsearch] Installing elasticsearch template to \_template/custom-abc-template  
[2019-09-18T15:15:09,867][INFO][logstash.outputs.elasticsearch] Installing elasticsearch template to \_template/custom-def-template  
[2019-09-18T15:15:09,958][INFO][logstash.outputs.elasticsearch] Using Index lifecycle management - this feature is currently in beta.  
[2019-09-18T15:15:09,987][INFO][logstash.outputs.elasticsearch] Using Index lifecycle management - this feature is currently in beta.  
[2019-09-18T15:15:09,988][WARN][logstash.outputs.elasticsearch] Overwriting supplied index name with rollover alias logstash  
[2019-09-18T15:15:09,989][WARN][logstash.outputs.elasticsearch] Overwriting supplied index name with rollover alias logstash  
[2019-09-18T15:15:10,029][INFO][logstash.outputs.elasticsearch] Creating rollover alias   
[2019-09-18T15:15:10,031][INFO][logstash.outputs.elasticsearch] Creating rollover alias

==\> /var/log/elasticsearch/elasticsearch.log \<==  
[2019-09-18T15:15:10,300][WARN][r.suppressed] [3Qz6On0] path: /%3Clogstash-500001%3E, params: {index=}  
java.lang.IllegalStateException: alias [logstash] has more than one write index [logstash-000001,logstash-500001]  
at org.elasticsearch.cluster.metadata.AliasOrIndex$Alias.computeAndValidateWriteIndex(AliasOrIndex.java:173) ~[elasticsearch-6.6.1.jar:6.6.1]  
at org.elasticsearch.cluster.metadata.MetaData$Builder.lambda$buildAliasAndIndexLookup$1(MetaData.java:1158) ~[elasticsearch-6.6.1.jar:6.6.1]

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 16, 2019, 2:28pm UTC](https://discuss.elastic.co/t/ilm-policy-overriden-by-logstash/200031/2 "2019-10-16T14:28:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.