We're using Elastic 7.17. What I'm trying to accomplish is: I want to ship logs using filebeat so that they land in a datastream of a specific name and with a specific lifecycle policy. The goal is to separate data from different sources, or of different types, for example logs from our apps should be separate from logs from Elastic itself, the same with metrics data.
How I currently understand the situation. Index templates define things like fields or number of shards, but can also link to lifecycle policies. Lifecycle policies define things like when to do a rollover or when to move an index from "hot" phase to "warm" phase. I think I understand those two. Where I'm getting lost is, how to configure it on filebeat side.
Somehow, the idea I have in head is, that I should be configuring as little as possible on filebeat side, and control everything on Elastic/cluster side. On filebeat side configuing things like setup.ilm.policy_name makes little sense to me. Policy itself has nothing to do with index names or if we're talking about indices or datastreams. It's index templates that link to polices, not the other way round. On the other hand, the documentation clearly states that "If index lifecycle management is enabled (which is typically the default), setup.template.name and setup.template.pattern are ignored.". So how does filebeat or Elastic choose which index template to use? Purely based on priority? I somehow cannot confirm that.
So I tried to ignore policies and configure filebeat with
setup:
ilm.enabled: false
template:
name: pc24-stream-logs
pattern: pc24-stream-logs
I created index template pc24-stream-logs manually with priority 200, configured it with patterns pc24-stream-logs-*, marked it as data stream, started filebeat and imported one log entry through it. Unfortunately, this entry landed in index filebeat-7.17.28-2025.03.03 that is not a data stream.
What am I missing? What am I doing wrong? To make the question a bit more specific, I did configure things like described here: Set up a data stream | Elasticsearch Guide [7.17] | Elastic. This page, unfortunately, doesn't say anything about configuring filebeat. How would I now need to configure filebeat so that it would send the data to datastream created there?