Need help with ILM when using logstash

[SimonR (Simon)]

Hi all, i am struggling to setup ILM when sending logs from beats to via Logstash to Elasticsearch.
Im looking to do daily indices, with rollover like the following index naming scheme:

"companyname-winlogbeat-7.7.0--00001" using only "companyname" as a hardcoded value in logstash and the rest coming from metadata and ILM.

I have exported the winlogbeat index templates to json, and edited the index template and index names inside the json to match the new index name, instead of just winlogbeat.
But the ILM does not happen, i can see the ilm policies, but they are not linked to the indices even thought i defined the index pattern.

My logstash output looks the following
Am i on the correct path, or am i going in the wrong direction?

        output {
  if "COMPANYNAME" in [tags] {
    elasticsearch {
      ssl => true
      cacert => '/etc/logstash/certs/ca.crt'
      manage_template => true
      template => "/etc/logstash/index_templates/COMPANYNAME-winlogbeat-7.7.0.json"
      template_name => "COMPANYNAME-winlogbeat-7.7.0"
      template_overwrite => true
      hosts => ["es-1.fqdn.com:9200", "es-2.fqdn.com:9200", "es-3.fqdn.com:9200"]
      user => "logstash_writer"
      password => "PASSWORD"
    }
  }

[rugenl (Len Rugen)]

Some things when I worked thru this issue sometime, but there have been a few releases since then.

1. I don't see the index name or ilm_rollover_alias in your config. You need to use index, ilm_rollover_alias doesn't work with %{[some_field][sub_field]}

2. You will need to create all templates and ILM policies manually, the "setup" of logstash won't create all possible combinations of your field values. (Called 'bootstraping' in the manual)

3. We have it working and it is very powerful, but setup will get you an advanced degree in ILM