Can't assign ilm policy to indexes created by logstash

[key]

Hi,
I'm using Filebeat, Logstash, Elasticsearch and Kibana all together to log data from an app.
For all these I use versions 7.2.0.
My problem is : I can't assign automatically an ilm policy to the indexes created by logstash.

For the pipeline here are the confs :
elasticsearch.yml

cluster.name: "docker-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
discovery.type: single-node

logstash.yml

http.host: "0.0.0.0"
path.config: /usr/share/logstash/pipeline

logstash.conf

input {
  beats {
    port => 5044
  }
}


filter{
  grok {
    match => { "message" => "%{LOGLEVEL:level} %{SPACE} %{TIMESTAMP_ISO8601:date} \[(?<thread>[^\]]+)\] %{JAVACLASS:class}  - %{GREEDYDATA:msg}" } 
  }
}


output {
	elasticsearch {
		hosts => ["elasticsearch:9200"]
    index => "%{[@metadata][beat]}-%{[fields][origin]}-%{+YYYY.MM.DD}"
    template_name => "logstash"
    ilm_policy => "log_policy"
    ilm_enabled => true
  }
}

kibana.yml

server.name: kibana
server.host: "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.ilm.enabled: true

filebeat.yml

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/application/applog.log
  fields:
    origin: app

output.logstash:
  hosts: ["logstash:5044"]

I have created the log_policy in Kibana. And assigned it to the logstash template.
In /_template/logstash I can see

"settings": {
"index": {
"lifecycle": {
"name": "log_policy"
},
"number_of_shards": "1",
"refresh_interval": "5s"
}
},

I checked _ilm/status, it returns

{
"operation_mode": "RUNNING"
}

but /_ilm/explain returns

"filebeat-app-2019.07.186": {
"index": "filebeat-app-2019.07.186",
"managed": false
},

How can I change this managed property to true for all new indexes created by logstash ?

[key]

Well, changing the index_patterns filed in the template make it work, but then I'm not sure that I shall keep all the others configurations, in particular in logstash.conf.
I mean this :

    template_name => "logstash"
    ilm_policy => "log_policy"
    ilm_enabled => true

[jsvd (João Duarte)]

@key

My problem is : I can't assign automatically an ilm policy to the indexes created by logstash.

At startup, and if ilm_enabled is true, logstash will add the ILM settings to the template, create "logstash-%{now}-00001" index with awrite alias ("logstash") and the rotation policy.

If you need to setup additional indices with the same policy you need to do them in elasticsearch directly.

[rugenl (Len Rugen)]

I think your template is missing an "index_patterns" option matching the index you are creating and the logstash index => will need to point to the write alias.

FYI, I'm doing this in 6.7.1 and my first try looked a lot like yours. I couldn't get the date to change on index rollover, it uses the original date. Doc says logstash ilm_pattern defaults to {now/d}-000001 but it didn't work for me.

[key]

Well I have put xpack.ilm.enabled: true in kibana.yml, is this what you meant by

@jsvd

ilm_enabled is true

?
Logstash didn't add the ilm setting (are we talking about the settings in logstash.conf ?) to the template, but I did manually and now it's working.

[key]

I don't think what I'm trying to do can be done with a version of elastic prior to 7.xx.
I don't either think I miss the template beacause /_template/logstash returns a template with th correct ilm policy.
Regarding the names of indexes, I set it using logstash.con, I don't know exactly at this moment what happens if the index is rolled over because it's too big but i'll see