Hi,
I use logstash to handle ILM configuration on my indexes. This works well on startup of Logstash when no index exists. An index is created using the rollover_alias with the default ilm pattern and the policy gets attached, all as expected.
However, when I delete the index in es while Logstash is running it creates a new index using the rollover_alias without default ilm pattern and no ilm policy is attached.
Am I missing something?
I run es and logstash version 7.1.1
elasticsearch {
id => write_to_es
hosts => "${ELASTICSEARCH_URL:http://elasticsearch:9200}"
template => "/usr/share/logstash/config/templates/flowlog.template.json"
template_name => "${NAMESPACE:namespace_not_set}-flowlog-v1"
template_overwrite => "true"
user => "${ELASTIC_USER:elastic}"
password => "${ELASTIC_PASSWORD:secret}"
ilm_enabled => "${ILM_ENABLED:true}"
ilm_policy => "${NAMESPACE:namespace_not_set}-flowlog"
ilm_rollover_alias => "${NAMESPACE:namespace_not_set}-flowlog-v1"
}
before index is deleted in ES:
GET /test4-flowlog-v1*/_settings
{
"test4-flowlog-v1-2019.06.24-000001" : {
"settings" : {
"index" : {
"lifecycle" : {
"name" : "test4-flowlog",
"rollover_alias" : "test4-flowlog-v1"
},
"codec" : "best_compression",
"mapping" : {
"total_fields" : {
"limit" : "5000"
}
},
"refresh_interval" : "10s",
"number_of_shards" : "2",
"provided_name" : "<test4-flowlog-v1-{now/d}-000001>",
"creation_date" : "1561403226974",
"number_of_replicas" : "1",
"uuid" : "WNhPkp2ZR7i5o8c7I8PhWw",
"version" : {
"created" : "7010199"
}
}
}
}
}
After index is deleted in ES:
GET /test4-flowlog-v1*/_settings
{
"test4-flowlog-v1" : {
"settings" : {
"index" : {
"creation_date" : "1561401486129",
"number_of_shards" : "1",
"number_of_replicas" : "1",
"uuid" : "oeREy6r9TcW658ozfyTI_A",
"version" : {
"created" : "7010199"
},
"provided_name" : "test4-flowlog-v1"
}
}
}
}
Update: I noticed that this behaviour was the result of the auto index creation being enabled in es. When I disabled auto index creation and repeated the above steps I noticed in the Logstash log that all events were failing because the index was not found. What I would like logstash to do in such and event is to re-create the index with the template and ilm settings or, second best, reload the pipeline that will force the index to be re-created. Is this possible?
Thanks,
Dennis