# Matching Logstash and Elasticsearch data stream configuration

**URL:** https://discuss.elastic.co/t/matching-logstash-and-elasticsearch-data-stream-configuration/281323
**Category:** Elasticsearch
**Tags:** datastreams
**Created:** [August 13, 2021, 8:58am UTC](https://discuss.elastic.co/t/matching-logstash-and-elasticsearch-data-stream-configuration/281323 "2021-08-13T08:58:02Z")
**Posts on this page:** 4
**Page:** 1

<div class="post-metadata">

### Author: ![jze1](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jze1](https://discuss.elastic.co/u/jze1)
#### Post date: [August 13, 2021, 8:58am UTC](https://discuss.elastic.co/t/matching-logstash-and-elasticsearch-data-stream-configuration/281323/1 "2021-08-13T08:58:02Z")

</div>

Hello,

can someone shed some light into configuration of logstash/data streams?

I have a bunch of apps (kubernetes pods) and I need to store logs of some for one year and the rest just for one week. I found out that by setting %{data\_stream.type}, %{data\_stream.dataset}, %{data\_stream.namespace} fields I can select the name of the data stream to route the logs to.

So I need to have at least two data streams: one with one week log retention and one with one year retention.

Now how would I define the properties of the data streams on Elasticsearch side? As per documentation the data stream requires some index template. WTF? So what would I use as index pattern and why? In the Logstash configuration I do not specify (I even CAN'T) the index(es) the logs would be routed to. When I specify index =\> '....' in the output section together with data\_stream =\> 'true' Logstash won't even start due to configuration error complaining about the index being specified.

---

<div class="post-metadata">

### Author: ![jze1](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jze1](https://discuss.elastic.co/u/jze1)
#### Post date: [August 13, 2021, 11:02am UTC](https://discuss.elastic.co/t/matching-logstash-and-elasticsearch-data-stream-configuration/281323/2 "2021-08-13T11:02:18Z")

</div>

OK, I got it. I have to create index template matching indices `<type>-<dataset>-<namespace>`, so in my case I create one index template named `logs-year-<namespace>` which will match `logs-year-<namespace>*` index pattern and then `logs-week-<namespace>` which will match `logs-*-<namespace>*` with lower priority than the `logs-year-<namespace>` template. In the Logstash filter I set `[data_stream][dataset]` field to 'week' or 'year' based on the required retention for the log entry.

---

<div class="post-metadata">

### Author: ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [August 15, 2021, 10:44pm UTC](https://discuss.elastic.co/t/matching-logstash-and-elasticsearch-data-stream-configuration/281323/3 "2021-08-15T22:44:40Z")

</div>

Welcome to our community and thanks for sharing your solution 😃

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 12, 2021, 10:45pm UTC](https://discuss.elastic.co/t/matching-logstash-and-elasticsearch-data-stream-configuration/281323/4 "2021-09-12T22:45:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.