Custom Index Name for Apache Module, but using ILM as well

[dsdameron]

I need to setup a Linux system running Apache to send logs into Elastic Search via Filebeat, but have the Apache module send to a different index. This custom index name needs to use the Filebeat/apache module mappings, pipelines, and dashboards. It will also need a custom Index Life Cycle as the Apache logs will have a different retention time.

I'm trying to figure out how to correctly do this. I've tagged events coming from the module with the apache_log tag, and I found this helpful article:

https://discuss.elastic.co/t/filebeat-apache-module-change-index-name/176955

but I can't use indices when using ILM. I'm not sure how to customize the index name for Apache, specify the ILM i defined within elastic search for Apache, and still send everything else into FileBeat's default. At what point in the configuration and how should this be defined?

Also, from playing around thus far, I'm wondering how much manual work is needed to keep this going. Do I need to manually duplicate the template and modify the index pattern and ILM to do this? And each time I apply each update? Or can this be set in Filebeat and Filebeat does it for you on 'filebeat setup' and filebeat updates? Thank you for your help.

[rugenl (Len Rugen)]

I use logstash, but in a similar way. If you do the ILM setup manually, it should work. You need to create the first index with a put, writing to the alias. See this

In the filebeat module set the output to the alias name.

[dsdameron]

Thank you Len. I am using FileBeat's Apache module, and it most helpfully does that first put for me, mange the template, dashboard etc. For any reading, this is the configuration I am using:

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat-apache-%{[agent.version]}"
setup.ilm.pattern: "{now/h}-000001" 
setup.ilm.policy_name: "filebeat-linux-apache"

The filebeat-linux-apache life cycle was defined in ElasticSearch.

Unfortunately, at this time it is not possible to say I want this module to send data to this index with this ilm, and this module to send to that index with that ILM. Each filebeat instance using ILM can only send to one index with one ILM at a time. More detailed info is here:

https://github.com/elastic/beats/issues/11866#issuecomment-589936554