Workaround to Dynamic Index Names When Ussing ILM Policy

I apologize if this has been asked a lot, but I did a lot of reading and I'm still not quite clear. For instance:

https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#_writing_to_different_indices_best_practices
https://github.com/logstash-plugins/logstash-output-elasticsearch/issues/858

In my current output I have this:

elasticsearch {
            hosts => ["https://mind.your.business:9200"]            
			manage_template => true
            template => "/etc/logstash/templates/outputs/business-core.json"
            template_overwrite => true
            template_name => "business-core"
            ilm_rollover_alias => "business-core"
            ilm_pattern => "000001"
            ilm_policy => "business-ilmpolicy"
            http_compression => true
            document_id => "%{[@metadata][document_id]}"
}

I will have 3 root index's, for example business, businessWealth, businessHappiness.

  • business-core-* (Index alias would be simply business-core)
  • businessWealth-core-*
  • businessHappiness-core-*

To make this work I would need 3 if clauses with in my output as I can not use dynamic variables here.

Reading the documentation it sounds like the ILM settings are used mainly for creating/updating the settings in ES.

Can I simply modify this setting above to:

elasticsearch {
            hosts => ["https://mind.your.business:9200"]            
			index => %{business}-core
			http_compression => true
            document_id => "%{[@metadata][document_id]}"
}

If I create all the templates outside of Logstash(I would use Ansible to post the template name in for the various combinations) does Logstash need to understand that ILM is even enabled? Or does it just need to be pointed to the correct index alias name and ES will
handle all the rollovers?

If I didn't want to create the ILM settings externally, could I just create another logstash dummy pipeline that would just be a long list of outputs with
all the various ILM settings. This would make my output sections a lot cleaner.

You can use your last example as long as all the ILM is configured in Elasticsearch and you are writing to the ILM write alias.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.