Datastream behavior in filebeat?

A very simple filebeat.yml configuration:

filebeat:
  inputs:
  - type: filestream
    id: vouchers-logs-stream
    paths:
      - /path/to/logs/*.log
    json:
      keys_under_root: true
      add_error_key: true
      overwrite_keys: true
      message_key: message
    parsers:
      - ndjson:
          target: ""
          add_error_key: true
output:
  elasticsearch:
    hosts: [ "..." ]
    username: "..."
    password: "..."
    index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd}
setup:
  template:
    name: "voucher-app-logs"
    pattern: "voucher-app-logs*"
    overwrite: false
  ilm:
    enabled: true
    policy_name: "voucher-app-logs-lifecycle-policy"

This configuration creates the following:

  • A data stream called: voucher-app-logs-8.7.0-2023.04.19
  • A index called .ds-voucher-app-logs-8.7.0-2023.04.19-2023.04.19-000001
  • And at each day, a new datastream is created

My questions:

  • Why create a new datastream everyday?
  • How to change the name of the datastream?
  • How to create only one datastream for these index patterns instead of creating one new ds every day?

I have read the docs and found nothing about that. Perhaps I missed something?

You do not want to do that.

index: voucher-app-logs-%{[agent.version]}

.ds-.... creation is driven by the ILM policy and the backing indices are meant to be opaque... if ILM is not set daily then new .ds-... will not be created daily

Since your config is creating a new datastream every day (which you do not want to do) then you get a backing index every day

But filebeat do this by default. It creates one datastream per day.

When I say "ds", I mean "datastream". Is there anyway to create only one datastream instead of creating one each day? WIth my current config, a new datastream is created everyday, as well as a index.

I want to create only one datastream and then all the indicies create by filebeat to be a part of that datastream. Is this possible to achieve?

No... it is because you set the index name like this
index: voucher-app-logs-%{[agent.version]}-%{+yyyy.MM.dd}
..........................................^^^^^^^^^^^^^^^ <!- THIS is not correct

That says create a new datastream every day....

Set the index as I suggested and it will not

If you do not set the index name at all it will create a datastream

filebeat-8.7.0 no daily date

Filebeat does not create a data stream per day by default, its is doing it because you configured it to it.

Try what I suggested first... your understanding is not correct

index: voucher-app-logs-%{[agent.version]}

THEN you can control how often the underlying .ds-.... is created with the ILM policy

1 Like

Wow, that's nice. Then the datastream created by filebeat follows the pattern described in the index parameter?

I haven't found that in the docs, perhaps I missed something, will check that later. I thought that the index parameter was intended to configure the index name created, even when ilm was enabled.

Thank you Stephen, it was extremely helpful..

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.