# Datastream behavior in filebeat?

**URL:** <https://discuss.elastic.co/t/datastream-behavior-in-filebeat/330325>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 19, 2023, 5:27pm UTC](https://discuss.elastic.co/t/datastream-behavior-in-filebeat/330325 "2023-04-19T17:27:40Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![matheuscirillo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matheuscirillo/32/118538_2.png) [@matheuscirillo](https://discuss.elastic.co/u/matheuscirillo)\
**Post date:** [April 19, 2023, 5:27pm UTC](https://discuss.elastic.co/t/datastream-behavior-in-filebeat/330325/1 "2023-04-19T17:27:41Z")

</div>

A very simple `filebeat.yml` configuration:

```auto
filebeat:
  inputs:
  - type: filestream
    id: vouchers-logs-stream
    paths:
      - /path/to/logs/*.log
    json:
      keys_under_root: true
      add_error_key: true
      overwrite_keys: true
      message_key: message
    parsers:
      - ndjson:
          target: ""
          add_error_key: true
output:
  elasticsearch:
    hosts: ["..."]
    username: "..."
    password: "..."
    index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd}
setup:
  template:
    name: "voucher-app-logs"
    pattern: "voucher-app-logs*"
    overwrite: false
  ilm:
    enabled: true
    policy_name: "voucher-app-logs-lifecycle-policy"

```

This configuration creates the following:

- A data stream called: voucher-app-logs-8.7.0-2023.04.19
- A index called .ds-voucher-app-logs-8.7.0-2023.04.19-2023.04.19-000001
- And at each day, a new datastream is created

My questions:

- Why create a new datastream everyday?
- How to change the name of the datastream?
- How to create only one datastream for these index patterns instead of creating one new ds every day?

I have read the docs and found nothing about that. Perhaps I missed something?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 19, 2023, 8:00pm UTC](https://discuss.elastic.co/t/datastream-behavior-in-filebeat/330325/2 "2023-04-19T20:00:02Z")

</div>

> [@matheuscirillo](#):
>
> Why create a new datastream everyday?

You do not want to do that.

> [@matheuscirillo](#):
>
> How to change the name of the datastream?

` index: voucher-app-logs-%{[agent.version]}`

> [@matheuscirillo](#):
>
> How to create only one datastream for these index patterns instead of creating one new ds every day?

`.ds-....` creation is driven by the ILM policy and the backing indices are meant to be opaque... if ILM is not set daily then new `.ds-...` will not be created daily

Since your config is creating a **new** datastream every day (which you do not want to do) then you get a backing index every day

---

<div class="post-metadata">

**Author:** ![matheuscirillo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matheuscirillo/32/118538_2.png) [@matheuscirillo](https://discuss.elastic.co/u/matheuscirillo)\
**Post date:** [April 19, 2023, 9:08pm UTC](https://discuss.elastic.co/t/datastream-behavior-in-filebeat/330325/3 "2023-04-19T21:08:31Z")

</div>

> [@stephenb](#):
>
> You do not want to do that.

But filebeat do this by default. It creates one datastream per day.

> [@stephenb](#):
>
> `.ds-....` creation is driven by the ILM policy and the backing indices are meant to be opaque... if ILM is not set daily then new `.ds-...` will not be created daily
> 
> Since your config is creating a **new** datastream every day (which you do not want to do) then you get a backing index every day

When I say "ds", I mean "datastream". Is there anyway to create only one datastream instead of creating one each day? WIth my current config, a new datastream is created everyday, as well as a index.

I want to create only one datastream and then all the indicies create by filebeat to be a part of that datastream. Is this possible to achieve?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [April 19, 2023, 10:57pm UTC](https://discuss.elastic.co/t/datastream-behavior-in-filebeat/330325/4 "2023-04-19T22:57:52Z")

</div>

> [@matheuscirillo](#):
>
> But filebeat do this by default. It creates one datastream per day.

No... it is because you set the index name like this  
`index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd}`  
`..........................................^^^^^^^^^^^^^^^` \<!- THIS is not correct

That says create a new datastream every day....

Set the index as I suggested and it will not

If you do not set the index name at all it will create a datastream

`filebeat-8.7.0` no daily date

Filebeat does not create a data stream per day by default, its is doing it because you configured it to it.

Try what I suggested first... your understanding is not correct

`index: voucher-app-logs-%{[agent.version]}`

THEN you can control how often the underlying `.ds-....` is created with the ILM policy

---

<div class="post-metadata">

**Author:** ![matheuscirillo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/matheuscirillo/32/118538_2.png) [@matheuscirillo](https://discuss.elastic.co/u/matheuscirillo)\
**Post date:** [April 20, 2023, 11:32am UTC](https://discuss.elastic.co/t/datastream-behavior-in-filebeat/330325/6 "2023-04-20T11:32:11Z")

</div>

Wow, that's nice. Then the datastream created by filebeat follows the pattern described in the `index` parameter?

I haven't found that in the docs, perhaps I missed something, will check that later. I thought that the `index` parameter was intended to configure the index name created, even when ilm was enabled.

Thank you Stephen, it was extremely helpful..

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 18, 2023, 1:32pm UTC](https://discuss.elastic.co/t/datastream-behavior-in-filebeat/330325/7 "2023-05-18T13:32:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.