# Filebeat 8.0.0-rc2 dynamic data stream name

**URL:** <https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397>\
**Category:** Beats\
**Tags:** filebeat, datastreams\
**Created:** [February 6, 2022, 2:46pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397 "2022-02-06T14:46:39Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![WouterAA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wouteraa/32/101430_2.png) [@WouterAA](https://discuss.elastic.co/u/WouterAA)\
**Post date:** [February 6, 2022, 2:46pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397/1 "2022-02-06T14:46:40Z")

</div>

Hello, we use filebeat on a k8s cluster to store logs per namespace. On filebeat 7 we create an index per namespace. Filebeat 8.0.0-rc2 uses data streams by default, which we think is a more elegant solution. However it does not seem possible to create a data stream per namespace. Did I miss something?

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 7, 2022, 1:25pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397/2 "2022-02-07T13:25:17Z")

</div>

How do you create the indices per namespace? Data streams are special indices, so most of the things that work with regular indices should work with data streams, too.

---

<div class="post-metadata">

**Author:** ![WouterAA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wouteraa/32/101430_2.png) [@WouterAA](https://discuss.elastic.co/u/WouterAA)\
**Post date:** [February 8, 2022, 12:20pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397/3 "2022-02-08T12:20:09Z")

</div>

Basically we did something like:

```auto
indices:
- index: "prefix-%{[kubernetes.labels.key]}
  when:
    has_fields:
    - 'kubernetes.labels.key'

```

It does not seem possible to configure something similar with data stream (on 8.0.0-rc2).

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 8, 2022, 1:35pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397/4 "2022-02-08T13:35:23Z")

</div>

To me it seems that you are sending to regular indices still. That should be fine. Did you disable ILM?

Nevertheless, I will look into it.

---

<div class="post-metadata">

**Author:** ![WouterAA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wouteraa/32/101430_2.png) [@WouterAA](https://discuss.elastic.co/u/WouterAA)\
**Post date:** [February 8, 2022, 1:44pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397/5 "2022-02-08T13:44:22Z")

</div>

To be clear. I would like to send events to dynamic data streams. So something like:

```auto
data_streams:
- data_stream: "prefix-%{}
  when:
    has_fields:
    - 'key'

```

Oh and ILM is still enabled.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 8, 2022, 1:57pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397/6 "2022-02-08T13:57:02Z")

</div>

~~Got it. Unfortunately, I can reproduce your issue. Do you mind opening an issue on GH?~~  
Where did you put the indices config?

For me, it works like this:

```auto
output.elasticsearch:
  hosts: ["gfdsgds"]
  indices:
  - index: "sdfgdsf"
    when: ......

```

---

<div class="post-metadata">

**Author:** ![WouterAA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wouteraa/32/101430_2.png) [@WouterAA](https://discuss.elastic.co/u/WouterAA)\
**Post date:** [February 8, 2022, 3:57pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397/7 "2022-02-08T15:57:41Z")

</div>

I did the same thing as you did on 8.0.0-rc2 with a %{} as index name. This created indices for me. Are you sure it creates data streams? I can / will recheck my configuration this evening.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [February 8, 2022, 4:53pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397/8 "2022-02-08T16:53:14Z")

</div>

Ah, I see what you mean now! Filebeat only creates data streams for the default indices e.g. filebeat-8.0.0. If you want to create a data stream for your custom indices, you have to specify it in their index templates by adding `"data_stream": {}` to it. Alternatively, if you have an alias you can convert that to a data stream: [Migrate to data stream API | Elasticsearch Guide [8.0] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.0/indices-migrate-to-data-stream.html#indices-migrate-to-data-stream)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 8, 2022, 6:53pm UTC](https://discuss.elastic.co/t/filebeat-8-0-0-rc2-dynamic-data-stream-name/296397/9 "2022-03-08T18:53:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.