Is it possible to configure dynamic indices with ILM?

[sarahw]

We currently use a single filebeat config with ILM disabled to ingest logs from an IIS web application and IIS access/error logs using the IIS module. We have been using filebeat to generate daily indices, but now we would like to change over to using ILM. With our current setup we are able to use a single filebeat instance configured to send logs to different index patterns dynamically based on event fields added by filebeat, so our application logs go to filebeat-*-applogs-* and the IIS access/error logs go to filebeat-*-default-*.

Our current config looks like:

    filebeat.inputs:
    - type: log
      enabled: true

      paths:
        - E:\web\myapp\service1\logs\*
        - E:\web\myapp\service2\logs\*

      fields:
        type: applogs


    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

    setup.ilm.enabled: false

    setup.template.settings:
      index.number_of_shards: 2

    setup.kibana:
      host: "http://mykibana:5601"

    output.elasticsearch:
      hosts: ["http://myelastic1:9200","myelastic2:9200"]
      index: "filebeat-%{[agent.version]}-%{[fields.type]:default}-%{+yyyy.MM.dd}"
      pipeline: "applogs"

    processors:
      - add_host_metadata:
          when.not.contains.tags: forwarded

    setup.template.name: "filebeat-%{[agent.version]}-default"
    setup.template.pattern: "filebeat-%{[agent.version]}-default-*"
    setup.template.overwrite: true

    setup.dashboards.index: "filebeat-*-default-*"

You can see we are using fields.type to dynamically create the index name as described here: Configure the Elasticsearch output | Filebeat Reference [7.11] | Elastic.

However, when setting up ILM, the output.elasticsearch.index and setup.template fields are ignored and you must use setup.ilm.rollover_alias to change the index name. My first thought was to use fields.type in rollover_alias as follows:

    filebeat.inputs:
    - type: log
      enabled: true

      paths:
        - E:\web\myapp\service1\logs\*
        - E:\web\myapp\service2\logs\*

      fields:
        type: applogs

    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

    setup.template.settings:
      index.number_of_shards: 2

    setup.kibana:
      host: "http://mykibana:5601"

    output.elasticsearch:
      hosts: ["http://myelastic1:9200","myelastic2:9200"]
      pipeline: "applogs"

    processors:
      - add_host_metadata:
          when.not.contains.tags: forwarded

    setup.ilm.enabled: true
    setup.ilm.policy_name: "filebeat-applogs-policy"
    setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-%{[fields.type]:default}"
    setup.ilm.pattern: "{now/d}-000001"

But this does not work because setup.ilm.rollover_alias doesn't know about fields.type and so the indices are is always created as filebeat-*-default-*, which matches the wrong template for the application logs and ingest fails.

I have been reading the comments here [Docs] Note that output.elasticsearch.index setting is ignored with ILM enabled · Issue #11866 · elastic/beats · GitHub, which suggest that I need to create separate filebeat configs/instances for different log types. Is this the best solution? Is there another way to get filebeat to send records to different indices based on log type using a single config? I understand from reading the github thread that event fields can't be used in the setup namespace but adding the field to the event is really just a hack to get filebeat to direct different log types to different index patterns. I don't care if the fields.type field shows up in the event at all.

I think the removal of output.elasticsearch.index is the most confusing part of the way filebeat works with ILM. It seems like setup.ilm.rollover_alias and setup.ilm.policy_name are primarily intended to configure the templates that filebeat loads, but now setup.rollover_alias is also serving the dual purpose of telling filebeat what the output index should be. Perhaps the goal was to make things more foolproof, but it seems like things would be simpler if the output.elasticsearch.index setting was not ignored when ILM is enabled and setup settings weren't used for output.

[Wolfram_Haussig]

Hi,

You do not have to change your file war config at all: As you said, this config is primarily to configure ILM so you can skip this and configure and initialize the indices manually:

Tutorial: Automate rollover with ILM | Elasticsearch Reference [7.11] | Elastic

Basically, you create an ILM policy, add that to an existing index template (or create a new one) and create the initial write index.

The downside is that you have to know ahead of time which indices are required - which can be hard depending if the applications on your server change a lot.

Best regards
Wolfram

[sarahw]

Thanks for your reply. I see what you are saying. We could turn off ILM and manage everything manually, adding the ILM fields to the templates and bootstrapping the alias and initial backing index for each group of logs. After thinking about it, it seemed easier to let filebeat manage the bootstrapping and template for the built in modules, so we decided to go with the two config approach and run two filebeat instances per server. So far this is working okay although it does make our ansible playbook more complicated.

In case this is useful for anyone trying to solve a similar problem we are using the following two configs.

application logs:

    filebeat.inputs:
    - type: log
      enabled: true

      paths:
        - E:\web\myapp\service1\logs\*
        - E:\web\myapp\service2\logs\*

      # This field doesn't do anything anymore but we leave it in for consistency
      fields:
        type: applogs

    # We load the template separately with ansible
    setup.template.enabled: false

    setup.kibana:
      host: "http://mykibana:5601"

    output.elasticsearch:
      hosts: ["http://myelastic1:9200","myelastic2:9200"]
      pipeline: "applogs"

    processors:
      - add_host_metadata:
          when.not.contains.tags: forwarded

    logging.files.name: filebeat-applogs

    setup.ilm.enabled: true
    setup.ilm.policy_name: "filebeat-applogs-policy"
    setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-applogs"
    setup.ilm.pattern: "{now/d}-000001"

web server access/error logs:

    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false

    setup.template.settings:
      index.number_of_shards: 2

    setup.kibana:
      host: "http://mykibana:5601"

    output.elasticsearch:
      hosts: ["http://myelastic1:9200","myelastic2:9200"]

    processors:
      - add_host_metadata:
          when.not.contains.tags: forwarded

    logging.files.name: filebeat-default

    setup.ilm.enabled: true
    setup.ilm.policy_name: "filebeat-default"
    setup.ilm.rollover_alias: "filebeat-%{[agent.version]}-default"
    setup.ilm.pattern: "{now/d}-000001"