ILM policy not attached when index is deleted while Logstash is running

[dennis.rietvink]

Hi,

I use logstash to handle ILM configuration on my indexes. This works well on startup of Logstash when no index exists. An index is created using the rollover_alias with the default ilm pattern and the policy gets attached, all as expected.

However, when I delete the index in es while Logstash is running it creates a new index using the rollover_alias without default ilm pattern and no ilm policy is attached.

Am I missing something?

I run es and logstash version 7.1.1

        elasticsearch {
            id                  => write_to_es
            hosts               => "${ELASTICSEARCH_URL:http://elasticsearch:9200}"
            template            => "/usr/share/logstash/config/templates/flowlog.template.json"
            template_name       => "${NAMESPACE:namespace_not_set}-flowlog-v1"
            template_overwrite  => "true"
            user                => "${ELASTIC_USER:elastic}"
            password            => "${ELASTIC_PASSWORD:secret}"
	    ilm_enabled         => "${ILM_ENABLED:true}"
	    ilm_policy          => "${NAMESPACE:namespace_not_set}-flowlog"
            ilm_rollover_alias  => "${NAMESPACE:namespace_not_set}-flowlog-v1"
        }

before index is deleted in ES:

GET /test4-flowlog-v1*/_settings

{
  "test4-flowlog-v1-2019.06.24-000001" : {
    "settings" : {
      "index" : {
        "lifecycle" : {
          "name" : "test4-flowlog",
          "rollover_alias" : "test4-flowlog-v1"
        },
        "codec" : "best_compression",
        "mapping" : {
          "total_fields" : {
            "limit" : "5000"
          }
        },
        "refresh_interval" : "10s",
        "number_of_shards" : "2",
        "provided_name" : "<test4-flowlog-v1-{now/d}-000001>",
        "creation_date" : "1561403226974",
        "number_of_replicas" : "1",
        "uuid" : "WNhPkp2ZR7i5o8c7I8PhWw",
        "version" : {
          "created" : "7010199"
        }
      }
    }
  }
}

After index is deleted in ES:

GET /test4-flowlog-v1*/_settings

{
  "test4-flowlog-v1" : {
    "settings" : {
      "index" : {
        "creation_date" : "1561401486129",
        "number_of_shards" : "1",
        "number_of_replicas" : "1",
        "uuid" : "oeREy6r9TcW658ozfyTI_A",
        "version" : {
          "created" : "7010199"
        },
        "provided_name" : "test4-flowlog-v1"
      }
    }
  }
}

Update: I noticed that this behaviour was the result of the auto index creation being enabled in es. When I disabled auto index creation and repeated the above steps I noticed in the Logstash log that all events were failing because the index was not found. What I would like logstash to do in such and event is to re-create the index with the template and ilm settings or, second best, reload the pipeline that will force the index to be re-created. Is this possible?

Thanks,
Dennis