I apologize if this has been asked a lot, but I did a lot of reading and I'm still not quite clear. For instance:
https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_writing_to_different_indices_best_practices
https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/858
In my current output I have this:
elasticsearch {
hosts => ["https://mind.your.business:9200"]
manage_template => true
template => "/etc/logstash/templates/outputs/business-core.json"
template_overwrite => true
template_name => "business-core"
ilm_rollover_alias => "business-core"
ilm_pattern => "000001"
ilm_policy => "business-ilmpolicy"
http_compression => true
document_id => "%{[@metadata][document_id]}"
}
I will have 3 root index's, for example business, businessWealth, businessHappiness.
- business-core-* (Index alias would be simply business-core)
- businessWealth-core-*
- businessHappiness-core-*
To make this work I would need 3 if clauses with in my output as I can not use dynamic variables here.
Reading the documentation it sounds like the ILM settings are used mainly for creating/updating the settings in ES.
Can I simply modify this setting above to:
elasticsearch {
hosts => ["https://mind.your.business:9200"]
index => %{business}-core
http_compression => true
document_id => "%{[@metadata][document_id]}"
}
If I create all the templates outside of Logstash(I would use Ansible to post the template name in for the various combinations) does Logstash need to understand that ILM is even enabled? Or does it just need to be pointed to the correct index alias name and ES will
handle all the rollovers?
If I didn't want to create the ILM settings externally, could I just create another logstash dummy pipeline that would just be a long list of outputs with
all the various ILM settings. This would make my output sections a lot cleaner.