Configure ILM in filebeat

[gizem]

Hello,
I was using 7.0.1 version of filebeat before, and my index configuration line this:
setup.template.name: "index"
setup.template.pattern: "index-*"
output.elasticsearch:
index: "index-%{[fields.type]:other}-%{+yyyy.MM.dd}"

I upgraded the filebeat to version 7.9.2 and I want to use ILM. I tried this configuration and it does not work:
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "index-%{[fields.type]:other}"
setup.ilm.pattern: "{now/d}-000001"

How can I achieve this? I want to use "fields.type" in my index.

Regards,
Gizem

[Wolfram_Haussig]

Hi,

This does not work as described in this comment:

ILM and templates are global settings. Beats do not manage templates + policies for multiple indices and have no access to fields in events. In fact templates and ILM policies are setup before the first event is available.

The solution would be to manually bootstrap the indexes as described here:

• create ILM policy
• create index template for index-other and all other possible values

PUT _index_template/index-other
{
  "index_patterns": ["index-other-*"],                 
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 1,
      "index.lifecycle.name": "my_policy",      
      "index.lifecycle.rollover_alias": "index-other"    
    }
  }
}

• bootstrap initial indexes

PUT index-other-000001
{
  "aliases": {
    "index-other": {
      "is_write_index": true
    }
  }
}

Then, instead of configuring ILM in the Beats you just have to configure Beats to write to the created ILM alias:

output.elasticsearch:
  index: "index-%{[fields.type]:other}"

Best regards
Wolfram

[gizem]

Thank for reply @Wolfram_Haussig.
I want to ask another thing. I tried to use static rollover alias, and set a policy for it. But new index is not created. I am using ingest pipeline and not use logstash. Did it happen because of this?

[Wolfram_Haussig]

Using a static rollover alias should work:

setup.ilm.enabled: auto
setup.ilm.rollover_alias: "index"
setup.ilm.pattern: "{now/d}-000001"

Can you post the configuration you tried? Did you get any logs?

[gizem]

My configuration in filebeat:
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat"
setup.ilm.pattern: "{now/d}-000001"

For trying I update the filebeat lifecycle policy maximum index size 20 KB. The other options are default.
My index: filebeat-2020.10.22-000001. It is keep loading data even if it exceeds 20 KB. Should new index is created like filebeat-2020.10.22-000002?

Should I stop/start something or delete the index?

[gizem]

When I stop the filebeat, new index is created. It is very intresting and I don't understand why.
And the old index storage size 871kb. I set the policy 20kb.

What is it about @Wolfram_Haussig?

[Wolfram_Haussig]

Hi,

I guess you are running into the effect that ElasticSearch does not poll often enough if the index meets the policy criteria. You can change that as described here (Default is 10 minutes):

indices.lifecycle.poll_interval: 30s

[gizem]

@Wolfram_Haussig

indices.lifecycle.poll_interval: 30s

This is the solution. Thank you very much @Wolfram_Haussig