ILM and using one index per Filebeat module

[philraj]

Hi,

Is there any way (without disabling ILM) to have Filebeat use a different index for each enabled module? This was easy before ILM by adding conditions to the output.elasticsearch.indices:

setup.template.name: "logs"
setup.template.pattern: "logs-*"
output.elasticsearch:
    indices:
    - index: "logs-system-%{[agent.version]}-%{+yyyy.ww}"
      when.contains:
        fileset.module: system

But I'm having a very hard time finding information on how to accomplish something similar with ILM. Do I have to disable it to get this functionality?

[rugenl (Len Rugen)]

Ithink you can, I have Logstash using variables in the index name to write to different indices for each different value of that variable.

• you will probably have to have a template for each different index pattern (see example below)
• filebeat's setup probably won't do it
• you will have create the first index for each pattern before you write to it. doc
• if you want the create date in the index name, dothis

A sample ILM template. I use a higher priority, so it includes the default filebeat-%{{agent.version}} template, then adds ILM for this particular index pattern.

{
    "order" : 130,
    "index_patterns" : [
      "filebeat-7.3.2-zabbix-xx*"
    ],
    "settings" : {
      "index.lifecycle.name": "beat-retain-180",
      "index.lifecycle.rollover_alias": "filebeat-7.3.2-zabbix-xx"
    }
}

Sample output from logstash:

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"

This example if apache logs from a zabbix server. Filebeat uses the fields to set these variables:

fields:
  campus: xx
  app_id: zabbix

[philraj]

I'll take a look at this, thank you.