Can't index from logstash to Elasticsearch using ILM

[anirudhra10 (Anirudh Ramesh)]

Hi Logstash support team,

I am trying to configure the output section of logstash to send logs to the right index based on the type of log by using ILM.

Here is the relevant logstash conf section:

if [type] == "usprod-kafkaconnect" {
      elasticsearch {
        id => "usprod-kafkaconnect-output"
        hosts => <%= to_json($es_nodes) %>
        document_id => "%{[@metadata][fingerprint]}"
        ilm_enabled => true
        ilm_pattern => "000001"
        ilm_policy => "generic-rollover-policy"
        ilm_rollover_alias => "usprod-kafkaconnect"
      }
  } else if [type] == "usprod-support" {
      elasticsearch {
        id => "usprod-support-output"
        hosts => <%= to_json($es_nodes) %>
        document_id => "%{[@metadata][fingerprint]}"
        ilm_enabled => true
        ilm_pattern => "000001"
        ilm_policy => "generic-rollover-policy"
        ilm_rollover_alias => "usprod-support"
    }
  }

Logstash gives the following error and goes into restart mode:

An unexpected error occurred! {:error=>#<LogStash::ConfigurationError: The specified ILM policy generic-rollover-policy does not exist on your Elasticsearch instance>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:86:in `maybe_create_ilm_policy'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/ilm.rb:11:in `setup_ilm'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.1.0-java/lib/logstash/outputs/elasticsearch/common.rb:52:in `block in setup_after_successful_connection'"]}

However, if either the if or else-if condition is removed, logstash doesn't throw the above error and forwards logs to the correct index.
I have also unsuccessfully tried multiple conditions like above, where in each condition has a different ILM policy.

Is there anyway to associate the same ILM policy via logstash but for different index patterns?

[rugenl (Len Rugen)]

If your indexes, policies and templates already exist, try changing ilm_rollover_alias to index.

I think logstash has problems doing the ILM setup for complex configurations. I used variables like [type] in the index name prior to ILM. That still works, but I have to do the ILM setup manually for all possible indices that will be created.

[anirudhra10 (Anirudh Ramesh)]

No, templates don't exist for all the 100+ indices that are to be managed by ILM. Logstash actually creates a template if one doesn't exist.
Variable substitution isn't allowed in ilm related fields acc. to the docs.
This is a rather simple if-else-if conditional configuration that works if ILM fields are removed.

[rugenl (Len Rugen)]

Below is a sanitized version of one of our pipeline output sections (logstash-7.5.0). The parameters index and ilm_rollover_alias have a relationship. if both are specified, ilm_rollover_alias is used.

My index => always points to an alias.

This configuration requires that I manually create the first index for any possible indices to be used. One specific index is not rolled over or date managed, some filebeat events are sent to elastic ingest pipelines and the default action is to write to ILM indices based on variable subs.

Here is the date math doc if you want the creation date to be part of the index name. (I find it confusing that the create an ILM index suffixed with just -1, I alyways use -000001)

  if ([fields][app_id] == "xxxx") {
    elasticsearch {
      hosts => ["https://elastic:9200"]
      cacert => "/etc/logstash/certs/mycas.cer"
      user => "user"
      password => "password"
      sniffing => false
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"
    }
  }
  else if "use_ingest" in [tags] {
    elasticsearch {
      hosts => ["https://elastic:9200"]
      cacert => "/etc/logstash/certs/mycas.cer"
      user => "user"
      password => "password"
      sniffing => false
      manage_template => false
      pipeline => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fileset][module]}-%{[fileset][name]}-pipeline"
      ilm_enabled => true
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"
    }
  }
  else {
    elasticsearch {
      hosts => ["https://elastic:9200"]
      cacert => "/etc/logstash/certs/mycas.cer"
      user => "user"
      password => "password"
      sniffing => false
      manage_template => false
      ilm_enabled => true
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"
    }
  }

[anirudhra10 (Anirudh Ramesh)]

It makes sense what you are doing i.e., point index to an alias. But that would involve a lot of manual steps if I were to implement this for all the indices.

Also, This might be working for you because you are not using ilm_policy anywhere. Can you try using a policy in all your conditions and see if you get an error?

[anirudhra10 (Anirudh Ramesh)]

Any response from Elastic support community would be appreciated