Hello, we use filebeat on a k8s cluster to store logs per namespace. On filebeat 7 we create an index per namespace. Filebeat 8.0.0-rc2 uses data streams by default, which we think is a more elegant solution. However it does not seem possible to create a data stream per namespace. Did I miss something?
How do you create the indices per namespace? Data streams are special indices, so most of the things that work with regular indices should work with data streams, too.
Basically we did something like:
indices:
- index: "prefix-%{[kubernetes.labels.key]}
when:
has_fields:
- 'kubernetes.labels.key'
It does not seem possible to configure something similar with data stream (on 8.0.0-rc2).
To me it seems that you are sending to regular indices still. That should be fine. Did you disable ILM?
Nevertheless, I will look into it.
To be clear. I would like to send events to dynamic data streams. So something like:
data_streams:
- data_stream: "prefix-%{}
when:
has_fields:
- 'key'
Oh and ILM is still enabled.
Got it. Unfortunately, I can reproduce your issue. Do you mind opening an issue on GH?
Where did you put the indices config?
For me, it works like this:
output.elasticsearch:
hosts: ["gfdsgds"]
indices:
- index: "sdfgdsf"
when: ......
I did the same thing as you did on 8.0.0-rc2 with a %{} as index name. This created indices for me. Are you sure it creates data streams? I can / will recheck my configuration this evening.
Ah, I see what you mean now! Filebeat only creates data streams for the default indices e.g. filebeat-8.0.0. If you want to create a data stream for your custom indices, you have to specify it in their index templates by adding "data_stream": {} to it. Alternatively, if you have an alias you can convert that to a data stream: Migrate to data stream API | Elasticsearch Guide [8.0] | Elastic
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.