Manage several data stream(s) in the elasitcsearch output with interpolation

[Pascal_Nuccio]

If you need to manage several data streams for one LOGSTASH instance, you can configure the elasticsearch output like this in your logstash configuration:

We must use a filter to configure the data_stream parameters (type, dataset and namespace)

filter: |-
  mutate {
    add_field => {
      "[data_stream][type]" => "logs"
      "[data_stream][dataset]" => "filebeat-lexi-%{[env_name]}"
      "[data_stream][namespace]" => "%{[@metadata][version]}"
    }
  }


output: |-
      elasticsearch {
            data_stream => true

        }

[Ugo_Sangiorgi]

Badly formatted index, after interpolation still contains placeholder

Just to give context, in case you are facing the error above, its likely because Elasticsearch output does not accept field names to compose the data_stream name like it does for a normal "index" name.

This will not work, for instance:

data_stream => true
data_stream_type => "logs"
data_stream_dataset => "%{[my_field]}"
data_stream_namespace => "%{[@metadata][version]}"

You should instead add the fields in a mutate filter, in the filter section, like @Pascal_Nuccio mentioned:

mutate {
    add_field => {
      "[data_stream][type]" => "logs"
      "[data_stream][dataset]" => "%{[my_field]}"
      "[data_stream][namespace]" => "%{[@metadata][version]}"
    }
  }