ILM parameters in config

[torten (Dan)]

Good day,

I want to set up ILM rollover_alias and policy_name parameters from the event fields, but no luck, all I got is errors like:

`ERROR        instance/beat.go:878        Exiting: failed to read the ilm rollover alias: key not found`

Config example:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/app/*.log
    json.keys_under_root: true
    json.add_error_key: true
    scan_frequency: 1s
    fields:
      rollover_alias: "infra-application"
      policy_name: "2_shards_30_days"

output.elasticsearch:
  enabled: true
  hosts: ["http://host1:9200", "http://host2:9200"]

setup.ilm.enabled: true
setup.ilm.rollover_alias: "%{[fields.rollover_alias]}"
setup.ilm.pattern: "{now/d}-000001"
setup.ilm.policy_name: "%{[fields.policy_name]}"

[shaunak (Shaunak Kashyap)]

Hi @torten,

Which version of Filebeat are you running?

Shaunak

[torten (Dan)]

Sorry for didn't mention it. 7.4.2 with basic licence and Elasticsearch with the same version.

[shaunak (Shaunak Kashyap)]

Thanks. Reading through the code, it looks like only certain fields may be used in the setup.ilm.rollover_alias and setup.ilm.policy_name settings: beat.name, beat.version, agent.name, agent.version, observer.name, observer.version. Of these the first two — beat.name and beat.version are only available for backwards compatibility and must not be used; use agent.name and agent.version instead.

So any fields other than the "global" ones mentioned above are not available in the format string used in setup.ilm.rollover_alias and setup.ilm.policy_name settings.

In your configuration I see you have fields.rollover_alias and fields.policy_name which are defined under the input configuration. These fields will get added to each Beat event as it is created by the input. However, the ILM setup can happen when a user runs filebeat setup or filebeat setup --index-management. When one of these commands is run Filebeat does not read data from inputs, so the fields configured in the filebeat.inputs section don't come into play at all.

Shaunak

[torten (Dan)]

Thanks for the explanation! I believe in my case the best option will be to send event to Logstash and then process it accroding to logic patterns.