Hi there, we are currently using a single, generic "logstash-*" ElasticSearch output which works quite well with the default ILM defaults of 30 days or 50GB and the automatic alias naming convention of logstash-2020.04.03-000001, logstash-2020.04.13-000002, logstash-2020.04.24-000003, etc.
We are currently adding different data types via additional pipelines in a single logstash process but would like to keep those different data types for different amounts of time. I was thinking of creating multiple indexes with different names using a mutate like:
mutate { add_field => { "[@metadata][target_index]" => "prod-%{+YYYY.MM.dd}" } }
but then I saw the comment here: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_writing_to_different_indices_best_practices
You cannot use dynamic variable substitution when ilm_enabled is true and when using ilm_rollover_alias .
How can I enable custom logstash indices and have the associated ILM policies like: logstash-custom1-2020.04.24-000003?