Datastream behavior in filebeat?

[matheuscirillo (Matheus C.)]

A very simple filebeat.yml configuration:

filebeat:
  inputs:
  - type: filestream
    id: vouchers-logs-stream
    paths:
      - /path/to/logs/*.log
    json:
      keys_under_root: true
      add_error_key: true
      overwrite_keys: true
      message_key: message
    parsers:
      - ndjson:
          target: ""
          add_error_key: true
output:
  elasticsearch:
    hosts: [ "..." ]
    username: "..."
    password: "..."
    index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd}
setup:
  template:
    name: "voucher-app-logs"
    pattern: "voucher-app-logs*"
    overwrite: false
  ilm:
    enabled: true
    policy_name: "voucher-app-logs-lifecycle-policy"

This configuration creates the following:

• A data stream called: voucher-app-logs-8.7.0-2023.04.19
• A index called .ds-voucher-app-logs-8.7.0-2023.04.19-2023.04.19-000001
• And at each day, a new datastream is created

My questions:

• Why create a new datastream everyday?
• How to change the name of the datastream?
• How to create only one datastream for these index patterns instead of creating one new ds every day?

I have read the docs and found nothing about that. Perhaps I missed something?

[stephenb (Stephen Brown)]

@matheuscirillo

Why create a new datastream everyday?

You do not want to do that.

@matheuscirillo

How to change the name of the datastream?

index: voucher-app-logs-%{[agent.version]}

@matheuscirillo

How to create only one datastream for these index patterns instead of creating one new ds every day?

.ds-.... creation is driven by the ILM policy and the backing indices are meant to be opaque... if ILM is not set daily then new .ds-... will not be created daily

Since your config is creating a new datastream every day (which you do not want to do) then you get a backing index every day

[matheuscirillo (Matheus C.)]

@stephenb

You do not want to do that.

But filebeat do this by default. It creates one datastream per day.

@stephenb

.ds-.... creation is driven by the ILM policy and the backing indices are meant to be opaque... if ILM is not set daily then new .ds-... will not be created daily

Since your config is creating a new datastream every day (which you do not want to do) then you get a backing index every day

When I say "ds", I mean "datastream". Is there anyway to create only one datastream instead of creating one each day? WIth my current config, a new datastream is created everyday, as well as a index.

I want to create only one datastream and then all the indicies create by filebeat to be a part of that datastream. Is this possible to achieve?

[stephenb (Stephen Brown)]

@matheuscirillo

But filebeat do this by default. It creates one datastream per day.

No... it is because you set the index name like this
index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd}
..........................................^^^^^^^^^^^^^^^ <!- THIS is not correct

That says create a new datastream every day....

Set the index as I suggested and it will not

If you do not set the index name at all it will create a datastream

filebeat-8.7.0 no daily date

Filebeat does not create a data stream per day by default, its is doing it because you configured it to it.

Try what I suggested first... your understanding is not correct

index: voucher-app-logs-%{[agent.version]}

THEN you can control how often the underlying .ds-.... is created with the ILM policy

[matheuscirillo (Matheus C.)]

Wow, that's nice. Then the datastream created by filebeat follows the pattern described in the index parameter?

I haven't found that in the docs, perhaps I missed something, will check that later. I thought that the index parameter was intended to configure the index name created, even when ilm was enabled.

Thank you Stephen, it was extremely helpful..