Reasoning behind attaching Lifecycle policies to index template

[trondhindenes]

We just updated to Elasticsearch 6.7, and were looking forward to start using Index Lifecycle Policies instead of Curator.

I was very surprised to see that index lifecycles are "activated" by attaching them to index templates, instead of simple index pattern matching. This makes the functionality a lot less valuable and harder to use, imho.

Couple of reasons:

• Index lifecycles will only apply to indexes created after the lifecycle was "activated" since templates are applied to indexes upon creation. This strikes me as a strange choice, as it requires the continued use of Curator (or whatever tool one's using) to manage old indexes, while new indexes are handled via policies.

• We have a lot of "prod/nonprod" indexes which are created from the same template, since the schema is identical. However, we want to have completely different retainment policies. This is not possible using policies, again since they are mapped to index templates and not indexes.

I was super-surprised to find this weird limitation. Am I missing something?

[trondhindenes]

reading a bit more here, it turns out that a policy can be applied to an index (https://www.elastic.co/guide/en/elasticsearch/reference/6.7/_switching_policies_for_an_index.html).

This means that we'll probably have to design out-of-band process to apply policies to indexes (especially since Kibana only seems to support attaching it at the template level).

So: nice that it is in fact possible to apply policies directly, but still I'm finding the design decisions strange. I'd love to be able to use this without having to run custom code outside the cluster to manage it.