# Can't assign ilm policy to indexes created by logstash

**URL:** <https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035>\
**Category:** Logstash\
**Created:** [July 5, 2019, 8:13am UTC](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035 "2019-07-05T08:13:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![key](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/key/32/45480_2.png) [@key](https://discuss.elastic.co/u/key)\
**Post date:** [July 5, 2019, 8:13am UTC](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035/1 "2019-07-05T08:13:48Z")

</div>

Hi,  
I'm using Filebeat, Logstash, Elasticsearch and Kibana all together to log data from an app.  
For all these I use versions 7.2.0.  
My problem is : I can't assign automatically an ilm policy to the indexes created by logstash.

For the pipeline here are the confs :  
_elasticsearch.yml_

```auto
cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
discovery.type: single-node

```

_logstash.yml_

```auto
http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline

```

_logstash.conf_

```auto
input {
  beats {
    port => 5044
  }
}

filter{
  grok {
    match => { "message" => "%{LOGLEVEL:level} %{SPACE} %{TIMESTAMP_ISO8601:date} \[(?<thread>[^\]]+)\] %{JAVACLASS:class} - %{GREEDYDATA:msg}" } 
  }
}

output {
	elasticsearch {
		hosts => ["elasticsearch:9200"]
    index => "%{[@metadata][beat]}-%{[fields][origin]}-%{+YYYY.MM.DD}"
    template_name => "logstash"
    ilm_policy => "log_policy"
    ilm_enabled => true
  }
}

```

_kibana.yml_

```auto
server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.ilm.enabled: true

```

_filebeat.yml_

```auto
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/application/applog.log
  fields:
    origin: app

output.logstash:
  hosts: ["logstash:5044"]

```

I have created the _log\_policy_ in Kibana. And assigned it to the _logstash_ template.  
In _/\_template/logstash_ I can see

```auto
"settings": {
"index": {
"lifecycle": {
"name": "log_policy"
},
"number_of_shards": "1",
"refresh_interval": "5s"
}
},

```

I checked _\_ilm/status_, it returns

```auto
{
"operation_mode": "RUNNING"
}

```

but _/\_ilm/explain_ returns

```auto
"filebeat-app-2019.07.186": {
"index": "filebeat-app-2019.07.186",
"managed": false
},

```

How can I change this _managed_ property to true for all new indexes created by logstash ?

---

<div class="post-metadata">

**Author:** ![key](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/key/32/45480_2.png) [@key](https://discuss.elastic.co/u/key)\
**Post date:** [July 5, 2019, 8:49am UTC](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035/2 "2019-07-05T08:49:53Z")

</div>

Well, changing the _index\_patterns_ filed in the template make it work, but then I'm not sure that I shall keep all the others configurations, in particular in _logstash.conf_.  
I mean this :

```auto
    template_name => "logstash"
    ilm_policy => "log_policy"
    ilm_enabled => true

```

---

<div class="post-metadata">

**Author:** ![jsvd](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsvd/32/6203_2.png) [@jsvd](https://discuss.elastic.co/u/jsvd)\
**Post date:** [July 8, 2019, 1:39pm UTC](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035/3 "2019-07-08T13:39:45Z")

</div>

> [@key](#):
>
> My problem is : I can't assign automatically an ilm policy to the indexes created by logstash.

At startup, and if ilm\_enabled is true, logstash will add the ILM settings to the template, create "logstash-%{now}-00001" index with awrite alias ("logstash") and the rotation policy.

If you need to setup additional indices with the same policy you need to do them in elasticsearch directly.

---

<div class="post-metadata">

**Author:** ![rugenl](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rugenl/32/12887_2.png) [@rugenl](https://discuss.elastic.co/u/rugenl)\
**Post date:** [July 8, 2019, 2:21pm UTC](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035/4 "2019-07-08T14:21:29Z")

</div>

I think your template is missing an "index\_patterns" option matching the index you are creating and the logstash index =\> will need to point to the write alias.

FYI, I'm doing this in 6.7.1 and my first try looked a lot like yours. I couldn't get the date to change on index rollover, it uses the original date. Doc says logstash ilm\_pattern defaults to {now/d}-000001 but it didn't work for me.

---

<div class="post-metadata">

**Author:** ![key](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/key/32/45480_2.png) [@key](https://discuss.elastic.co/u/key)\
**Post date:** [July 8, 2019, 2:37pm UTC](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035/5 "2019-07-08T14:37:38Z")

</div>

Well I have put _xpack.ilm.enabled: true_ in kibana.yml, is this what you meant by

> [@jsvd](#):
>
> ilm\_enabled is true

?  
Logstash didn't add the ilm setting (are we talking about the settings in _logstash.conf_ ?) to the template, but I did manually and now it's working.

---

<div class="post-metadata">

**Author:** ![key](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/key/32/45480_2.png) [@key](https://discuss.elastic.co/u/key)\
**Post date:** [July 8, 2019, 2:40pm UTC](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035/6 "2019-07-08T14:40:25Z")

</div>

I don't think what I'm trying to do can be done with a version of elastic prior to 7.xx.  
I don't either think I miss the template beacause _/\_template/logstash_ returns a template with th correct ilm policy.  
Regarding the names of indexes, I set it using _logstash.con_, I don't know exactly at this moment what happens if the index is rolled over because it's too big but i'll see 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 5, 2019, 2:40pm UTC](https://discuss.elastic.co/t/cant-assign-ilm-policy-to-indexes-created-by-logstash/189035/7 "2019-08-05T14:40:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.