Can't assign ilm policy to indexes created by logstash

I'm using Filebeat, Logstash, Elasticsearch and Kibana all together to log data from an app.
For all these I use versions 7.2.0.
My problem is : I can't assign automatically an ilm policy to the indexes created by logstash.

For the pipeline here are the confs :
elasticsearch.yml "docker-cluster"
discovery.zen.minimum_master_nodes: 1
discovery.type: single-node

logstash.yml ""
path.config: /usr/share/logstash/pipeline


input {
  beats {
    port => 5044

  grok {
    match => { "message" => "%{LOGLEVEL:level} %{SPACE} %{TIMESTAMP_ISO8601:date} \[(?<thread>[^\]]+)\] %{JAVACLASS:class}  - %{GREEDYDATA:msg}" } 

output {
	elasticsearch {
		hosts => ["elasticsearch:9200"]
    index => "%{[@metadata][beat]}-%{[fields][origin]}-%{+YYYY.MM.DD}"
    template_name => "logstash"
    ilm_policy => "log_policy"
    ilm_enabled => true

kibana.yml kibana "0"
elasticsearch.hosts: http://elasticsearch:9200
xpack.ilm.enabled: true


- type: log
  enabled: true
    - /var/log/application/applog.log
    origin: app

  hosts: ["logstash:5044"]

I have created the log_policy in Kibana. And assigned it to the logstash template.
In /_template/logstash I can see

"settings": {
"index": {
"lifecycle": {
"name": "log_policy"
"number_of_shards": "1",
"refresh_interval": "5s"

I checked _ilm/status, it returns

"operation_mode": "RUNNING"

but /_ilm/explain returns

"filebeat-app-2019.07.186": {
"index": "filebeat-app-2019.07.186",
"managed": false

How can I change this managed property to true for all new indexes created by logstash ?

Well, changing the index_patterns filed in the template make it work, but then I'm not sure that I shall keep all the others configurations, in particular in logstash.conf.
I mean this :

    template_name => "logstash"
    ilm_policy => "log_policy"
    ilm_enabled => true

At startup, and if ilm_enabled is true, logstash will add the ILM settings to the template, create "logstash-%{now}-00001" index with awrite alias ("logstash") and the rotation policy.

If you need to setup additional indices with the same policy you need to do them in elasticsearch directly.

I think your template is missing an "index_patterns" option matching the index you are creating and the logstash index => will need to point to the write alias.

FYI, I'm doing this in 6.7.1 and my first try looked a lot like yours. I couldn't get the date to change on index rollover, it uses the original date. Doc says logstash ilm_pattern defaults to {now/d}-000001 but it didn't work for me.

Well I have put xpack.ilm.enabled: true in kibana.yml, is this what you meant by

Logstash didn't add the ilm setting (are we talking about the settings in logstash.conf ?) to the template, but I did manually and now it's working.

I don't think what I'm trying to do can be done with a version of elastic prior to 7.xx.
I don't either think I miss the template beacause /_template/logstash returns a template with th correct ilm policy.
Regarding the names of indexes, I set it using logstash.con, I don't know exactly at this moment what happens if the index is rolled over because it's too big but i'll see :slight_smile:

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.