Logstash and filebeat data streams

[gborg (Gudrun Olafsdottir)]

Hello

I have a setup where I run filebeats, which send their logs to a queue, logstash reads off the queue and writes the data to elasticsearch.

With version 7 I had no issues, I recently upgraded to elasticsearch to elasticsearch 8 and was going to start updating the beats, where I ran into a problem

The default behaviours of beats (when connected directly to elasticsearch) is to write to the datastram "%{beat-name}-%{beat-version", "filebeat-8.4.1" for example

And i want to use the standard datastream format with my beats data but I can't since the logstash output only supports writing datastreams in the "%{type}-%{dataset}-%{namespace}" format, "logs-default-generic" for example

How can I use logstash and the standard datastreams together ?

[gborg (Gudrun Olafsdottir)]

I mangaged to solve it

by not setting the elasticsearch output as a datastream, specifying the action and the index name

 elasticsearch {
                        [.....]
                        action => "create"
                        index => "%{[@metadata][beat]}-%{[@metadata][version]}"
 }