Filebeat 8.0.0-rc2 dynamic data stream name

Elastic Stack Beats
,
1

Hello, we use filebeat on a k8s cluster to store logs per namespace. On filebeat 7 we create an index per namespace. Filebeat 8.0.0-rc2 uses data streams by default, which we think is a more elegant solution. However it does not seem possible to create a data stream per namespace. Did I miss something?

2

How do you create the indices per namespace? Data streams are special indices, so most of the things that work with regular indices should work with data streams, too.

3

Basically we did something like:

indices:
- index: "prefix-%{[kubernetes.labels.key]}
  when:
    has_fields:
    - 'kubernetes.labels.key'

It does not seem possible to configure something similar with data stream (on 8.0.0-rc2).

4

To me it seems that you are sending to regular indices still. That should be fine. Did you disable ILM?

Nevertheless, I will look into it.

5

To be clear. I would like to send events to dynamic data streams. So something like:

data_streams:
- data_stream: "prefix-%{}
  when:
    has_fields:
    - 'key'

Oh and ILM is still enabled.

6

Got it. Unfortunately, I can reproduce your issue. Do you mind opening an issue on GH?
Where did you put the indices config?

For me, it works like this:

output.elasticsearch:
  hosts: ["gfdsgds"]
  indices:
  - index: "sdfgdsf"
    when: ......
7

I did the same thing as you did on 8.0.0-rc2 with a %{} as index name. This created indices for me. Are you sure it creates data streams? I can / will recheck my configuration this evening.

8

Ah, I see what you mean now! Filebeat only creates data streams for the default indices e.g. filebeat-8.0.0. If you want to create a data stream for your custom indices, you have to specify it in their index templates by adding "data_stream": {} to it. Alternatively, if you have an alias you can convert that to a data stream: Migrate to data stream API | Elasticsearch Guide [8.0] | Elastic

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.