Need to understand difference between alias and rollover_alias while creating index template for ILM implementation to multiple environments

[adeel109 (Syed Adeel Ahmad)]

I have one ILM policy named haproxy-ilm for all haprox logs. There are 4 different indices to which logs from each environment are written, i.e., dev, prestaging, staging, and production.

Below is the configuration of logstash prior to ILM:

#filter.conf
filter {
  if [app_type] == "haproxy" {
    grok {
      patterns_dir => ["/etc/logstash/conf.d/patterns/haproxy"]
      match => ["message", "%{HAPROXYHTTP}"]
      add_field => { "[@metadata][index_prefix]" => "%{env}-%{app_type}" }
    }
  }
}

#output.conf
output {
  if [app_type] == "haproxy" and "_grokparsefailure" not in [tags] {
     elasticsearch {
      hosts => ["{{ es_servers }}"]
      index => "%{[@metadata][index_prefix]}"
     }
   }
}

While creating index templates for mapping with ILM policy, I thought I could create one template and have all the index patterns in it, like, "index_patterns": ["dev-haproxy-*", "prestaging-haproxy-*, ... ]. I did it using kibana Index Management.

After this, I could modify logstash output.conf like:

#output.conf
output {
  if [app_type] == "haproxy" and "_grokparsefailure" not in [tags] {
     elasticsearch {
      hosts => ["{{ es_servers }}"]
      ilm_rollover_alias => "%{[@metadata][index_prefix]}"
      ilm_policy => "haproxy-ilm"
     }
   }
}

Finally, bootstrap indices were created with below command for respective environments:

curl -X PUT "elastmaster.mydomain.net:9200/%3Cdev-haproxy-%7Bnow%2Fd%7D-000001%3E?pretty" -H 'Content-Type: application/json' -d'
{
  "aliases": {
    "dev-haproxy": {
      "is_write_index": true
    }
  }
}
'

When I started pushing logs, I got this error:
illegal_argument_exception: setting [index.lifecycle.rollover_alias] for index [dev-haproxy-2020.07.25-000001] is empty or not defined.

This is why, instead of having just one index template, I had to create 4 index templates for each environment and define their respective index.lifecycle.rollover_alias like:

curl -X PUT "elastmaster.mydomain.net:9200/_template/dev?pretty" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["dev-haproxy-*"],                 
  "settings": {
    "index.lifecycle.name": "haproxy-ilm",      
    "index.lifecycle.rollover_alias": "dev-haproxy"    
  }
}
'

Please help me understand how can this be optimised to use only one index template as haproxy is not the only log source. I'll have to create a lot of index templates if I go like this.