Hello,
I was using 7.0.1 version of filebeat before, and my index configuration line this:
setup.template.name: "index"
setup.template.pattern: "index-*"
output.elasticsearch:
index: "index-%{[fields.type]:other}-%{+yyyy.MM.dd}"
I upgraded the filebeat to version 7.9.2 and I want to use ILM. I tried this configuration and it does not work:
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "index-%{[fields.type]:other}"
setup.ilm.pattern: "{now/d}-000001"
How can I achieve this? I want to use "fields.type" in my index.
ILM and templates are global settings. Beats do not manage templates + policies for multiple indices and have no access to fields in events. In fact templates and ILM policies are setup before the first event is available.
The solution would be to manually bootstrap the indexes as described here:
create ILM policy
create index template for index-other and all other possible values
Thank for reply @Wolfram_Haussig.
I want to ask another thing. I tried to use static rollover alias, and set a policy for it. But new index is not created. I am using ingest pipeline and not use logstash. Did it happen because of this?
My configuration in filebeat:
setup.ilm.enabled: auto
setup.ilm.rollover_alias: "filebeat"
setup.ilm.pattern: "{now/d}-000001"
For trying I update the filebeat lifecycle policy maximum index size 20 KB. The other options are default.
My index: filebeat-2020.10.22-000001. It is keep loading data even if it exceeds 20 KB. Should new index is created like filebeat-2020.10.22-000002?
Should I stop/start something or delete the index?
When I stop the filebeat, new index is created. It is very intresting and I don't understand why.
And the old index storage size 871kb. I set the policy 20kb.
I guess you are running into the effect that ElasticSearch does not poll often enough if the index meets the policy criteria. You can change that as described here (Default is 10 minutes):
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.