# String inteprolation in logstash data\_stream fields
**URL:** https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556
**Category:** Logstash
**Created:** [October 26, 2022, 8:01pm UTC](https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556 "2022-10-26T20:01:36Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Austin\_ES\_Questions](https://discuss.elastic.co/u/Austin_ES_Questions)
#### Post date: [October 26, 2022, 8:01pm UTC](https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556/1 "2022-10-26T20:01:36Z")
How can I write to different data streams for different kinesis input? We are trying to add fields in the inputs and use string interpolation in the outputs to no avail.
We had been using index+ilm\_enabled in logstash, like this
In kinesis input:
```auto
add_field => { _meta_index => "api-events" }
```
in elasticsearch output:
```auto
index => "%{_meta_index}"
ilm_enabled => "true"
```
we are trying to migrate to using data streams, like this:
in kinesis input:
```auto
add_field => { _meta_data_stream_dataset => "api" }
add_field => { _meta_data_stream_namespace => "events" }
```
in elasticsearch output we've tried both:
```auto
data_stream => "true"
data_stream_type => "logs"
data_stream_dataset => "events-%{_meta_data_stream_dataset}"
data_stream_namespace => "%{_meta_data_stream_namespace}"
```
and
```auto
data_stream => "true"
data_stream_type => "logs"
data_stream_dataset => "events-%{[_meta_data_stream_dataset]}"
data_stream_namespace => "%{[_meta_data_stream_namespace]}"
```
but neither works, instead writing to
```auto
.ds-logs-events-%{_meta_data_stream_dataset}-%{_meta_data_stream_namespace}-2022.10.26-000001
```
and
```auto
.ds-logs-events-%{[_meta_data_stream_dataset]}-%{[_meta_data_stream_namespace]}-2022.10.26-000001
```
I know my fields are being set because I can see example in a record:
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [October 26, 2022, 8:16pm UTC](https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556/2 "2022-10-26T20:16:17Z")
I don't think this is supported, there was a [similar question](https://discuss.elastic.co/t/dynamic-naming-of-data-streams/283281) about it, and it seems that the data\_stream settings will not sprintf the value.
I do not use data streams, but from the elasticsearch output [documentation](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-data_stream_auto_routing) there is a setting named `data_stream_auto_routing` that may help you achieve what you want.
From what I understand, you will need to set this to `true` and create the following fields.
`data_stream.type`, `data_stream.dataset` and `data_stream.namespace`, then if those fields exist in the event, they will be used instead of the settings.
---
### Author:  [@Austin\_ES\_Questions](https://discuss.elastic.co/u/Austin_ES_Questions)
#### Post date: [October 26, 2022, 8:23pm UTC](https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556/3 "2022-10-26T20:23:34Z")
I understand the general vibe that there is an opinionated solution, I just don't know what it is.
Would an Elasticsearch team member be able to post a specific, working example? The docs have various references to dot notation and `[bracket]` notation, and it's unclear what the exact correct format is
---
### Author:  [@Austin\_ES\_Questions](https://discuss.elastic.co/u/Austin_ES_Questions)
#### Post date: [October 26, 2022, 8:41pm UTC](https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556/4 "2022-10-26T20:41:19Z")
None of these seem to be valid configurations
```auto
add_field => { data_stream => { dataset => "events_api" } }
add_field => { data_stream => { namespace => "events" } }
```
```auto
add_field => { data_stream.dataset => "events_api" } }
add_field => { data_stream.namespace => "events" } }
```
```auto
add_field => { [data_stream][dataset] => "events_api" } }
add_field => { [data_stream][namespace] => "events" } }
```
---
### Author:  [@Austin\_ES\_Questions](https://discuss.elastic.co/u/Austin_ES_Questions)
#### Post date: [October 26, 2022, 8:51pm UTC](https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556/5 "2022-10-26T20:51:53Z")
Doing both
```auto
input {
kinesis {
...
add_field => { _meta_data_stream_dataset => "api" }
add_field => { _meta_data_stream_namespace => "events" }
}
}
filter {
mutate {
add_field => { "[data_stream][dataset]" => "%{_meta_data_stream_dataset}" }
add_field => { "[data_stream][namespace]" => "%{_meta_data_stream_namespace}" }
}
}
```
works but is quite gross - is there a nice way to just set it in the input?
---
### Author:  [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [October 26, 2022, 9:00pm UTC](https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556/6 "2022-10-26T21:00:53Z")
> [@Austin\_ES\_Questions](#):
>
> The docs have various references to dot notation and `[bracket]` notation, and it's unclear what the exact correct format is
In Logstash when you want to work with a nested field, like `data_stream.type`, you need to refer to it as `[data_stream][type]`, if you use `data_stream.type` in logstash, it is a reference to a field with a literal dot in its name.
In Elasticsearch and Kibana to refer to the same nested field you just use `data_stream.type`.
The `[bracket][nested]` is only used in Logstash, and this can be confusing some times.
So, to add a nested field you just use:
```auto
add_field => { "[top-level][nested]" => "value" }`
```
> [@Austin\_ES\_Questions](#):
>
> works but is quite gross - is there a nice way to just set it in the input?
Just use the nested fields in the input.
```auto
input {
kinesis {
...
add_field => { "[data_stream][dataset]" => "api" }
add_field => { "[data_stream][namespace]" => "events" }
add_field => { "[data_stream][type]" => "logs" }
}
}
```
From the documentation, you need to add the 3 fields or it will fallback to the configuration in the output.
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 26, 2022, 9:42pm UTC](https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556/7 "2022-10-26T21:42:05Z")
> [@Austin\_ES\_Questions](#):
>
> ```auto
> index => "%{_meta_index}"
> ilm_enabled => "true"
>
> ```
If ilm\_enabled is true, then the index option is [overwritten](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/7c24cfa4c3ef76078a77f0d317038b2c7e0c44db/lib/logstash/outputs/elasticsearch/ilm.rb#L7), so that is not going to do what you want.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 23, 2022, 9:43pm UTC](https://discuss.elastic.co/t/string-inteprolation-in-logstash-data-stream-fields/317556/8 "2022-11-23T21:43:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.