We currently use a single filebeat config with ILM disabled to ingest logs from an IIS web application and IIS access/error logs using the IIS module. We have been using filebeat to generate daily indices, but now we would like to change over to using ILM. With our current setup we are able to use a single filebeat instance configured to send logs to different index patterns dynamically based on event fields added by filebeat, so our application logs go to filebeat-*-applogs-* and the IIS access/error logs go to filebeat-*-default-*.
Our current config looks like:
filebeat.inputs:
- type: log
enabled: true
paths:
- E:\web\myapp\service1\logs\*
- E:\web\myapp\service2\logs\*
fields:
type: applogs
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.ilm.enabled: false
setup.template.settings:
index.number_of_shards: 2
setup.kibana:
host: "http://mykibana:5601"
output.elasticsearch:
hosts: ["http://myelastic1:9200","myelastic2:9200"]
index: "filebeat-%{[agent.version]}-%{[fields.type]:default}-%{+yyyy.MM.dd}"
pipeline: "applogs"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
setup.template.name: "filebeat-%{[agent.version]}-default"
setup.template.pattern: "filebeat-%{[agent.version]}-default-*"
setup.template.overwrite: true
setup.dashboards.index: "filebeat-*-default-*"
You can see we are using fields.type to dynamically create the index name as described here: Configure the Elasticsearch output | Filebeat Reference [7.11] | Elastic.
However, when setting up ILM, the output.elasticsearch.index and setup.template fields are ignored and you must use setup.ilm.rollover_alias to change the index name. My first thought was to use fields.type in rollover_alias as follows:
filebeat.inputs:
- type: log
enabled: true
paths:
- E:\web\myapp\service1\logs\*
- E:\web\myapp\service2\logs\*
fields:
type: applogs
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 2
setup.kibana:
host: "http://mykibana:5601"
output.elasticsearch:
hosts: ["http://myelastic1:9200","myelastic2:9200"]
pipeline: "applogs"
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
setup.ilm.enabled: true
setup.ilm.policy_name: "filebeat-applogs-policy"
setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-%{[fields.type]:default}"
setup.ilm.pattern: "{now/d}-000001"
But this does not work because setup.ilm.rollover_alias doesn't know about fields.type and so the indices are is always created as filebeat-*-default-*, which matches the wrong template for the application logs and ingest fails.
I have been reading the comments here [Docs] Note that output.elasticsearch.index setting is ignored with ILM enabled · Issue #11866 · elastic/beats · GitHub, which suggest that I need to create separate filebeat configs/instances for different log types. Is this the best solution? Is there another way to get filebeat to send records to different indices based on log type using a single config? I understand from reading the github thread that event fields can't be used in the setup namespace but adding the field to the event is really just a hack to get filebeat to direct different log types to different index patterns. I don't care if the fields.type field shows up in the event at all.
I think the removal of output.elasticsearch.index is the most confusing part of the way filebeat works with ILM. It seems like setup.ilm.rollover_alias and setup.ilm.policy_name are primarily intended to configure the templates that filebeat loads, but now setup.rollover_alias is also serving the dual purpose of telling filebeat what the output index should be. Perhaps the goal was to make things more foolproof, but it seems like things would be simpler if the output.elasticsearch.index setting was not ignored when ILM is enabled and setup settings weren't used for output.