ILM and using one index per Filebeat module


Is there any way (without disabling ILM) to have Filebeat use a different index for each enabled module? This was easy before ILM by adding conditions to the output.elasticsearch.indices: "logs"
setup.template.pattern: "logs-*"
    - index: "logs-system-%{[agent.version]}-%{+yyyy.ww}"
        fileset.module: system

But I'm having a very hard time finding information on how to accomplish something similar with ILM. Do I have to disable it to get this functionality?

Ithink you can, I have Logstash using variables in the index name to write to different indices for each different value of that variable.

  • you will probably have to have a template for each different index pattern (see example below)
  • filebeat's setup probably won't do it
  • you will have create the first index for each pattern before you write to it. doc
  • if you want the create date in the index name, dothis

A sample ILM template. I use a higher priority, so it includes the default filebeat-%{{agent.version}} template, then adds ILM for this particular index pattern.

    "order" : 130,
    "index_patterns" : [
    "settings" : {
      "": "beat-retain-180",
      "index.lifecycle.rollover_alias": "filebeat-7.3.2-zabbix-xx"

Sample output from logstash:

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{[fields][app_id]}-%{[fields][campus]}"

This example if apache logs from a zabbix server. Filebeat uses the fields to set these variables:

  campus: xx
  app_id: zabbix

I'll take a look at this, thank you.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.