Filebeat and Data stream

[HelpComputer]

Hello,

What would be the best way to get the Filebeat legacy template converted into a data stream template or some other ILM? It seems like data streams are recommended based on this guide - https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html My goal is to get lifecycle management configured correctly with Filebeat so that when the indices reach a certain size or date they rollover or are deleted. I don’t want to lose all the mappings and index settings from the Filebeat template that’s included with Filebeat. If there’s another method that would be better, please share. I see there are built in data stream templates but none of them appear to have the Filebeat mappings.

Also, would I need to change anything on my Logstash output to get it to work correctly? Here’s my current output regarding index in Logstash.

index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"

Environment and Setup

Elasticsearch 7.9

Kibana 7.9

Filebeat 7.9.3

Logstash 7.9

Flow
Filebeat -> Logstash -> Elasticsearch

Thanks!

[warkolm (Mark Walkom)]

By default at the moment, Filebeat will use ILM.
That might change to data streams in the future, but we will make sure mappings etc are applied.

Given you are using Logstash that makes it a little bit harder. You will want to run the setup for Filebeat so that it bootstraps the ILM policy on Elasticsearch. Then tell Logstash to send to the Filebeat alias, ie don't use time based indices like you have there in the output.