Migrate to datastreams withou aliases + datastreams filter

[bilak (Lukáš Vasek)]

Hello,
I'd like to use data streams and migrate our indices to it. However currently we don't use aliases. Do I have to create an alias for given indices pattern (technical-logs*) at first and only after that I'm able to migrate to data streams?

Second question is more about filtering. When we are filtering by date we just get all indices names from ES and then we search in those indices which match the given filter (e.g. from technical-logs-20230101 to technical-logs-20230118). When we switch to data streams it seems like the indices will be hidden. So the question is am I able to somehow specify in which indices should query search? Or is it all automatically being applied when I specify @timestamp parameter?

Thanks

[warkolm (Mark Walkom)]

Welcome to our community!

@bilak

Do I have to create an alias for given indices pattern (technical-logs*) at first and only after that I'm able to migrate to data streams?

Generally you would reindex older data into the data stream so you don't need to worry about this.

@bilak

When we are filtering by date we just get all indices names from ES and then we search in those indices which match the given filter

It might be easier to provide a bit more context on how you are querying, because generally that's not what you'd do and is inefficient.

[bilak (Lukáš Vasek)]

@warkolm

Generally you would reindex older data into the data stream so you don't need to worry about this.

the question was more regarding this topic. Documentation tells only about migration from index alias. So if I have many indices (technical-logs-20230101, technical-logs-20230102, ...) which I want to migrate to one data stream am I able to do that without creating an alias for those indices? Currently there is no alias.

@warkolm

It might be easier to provide a bit more context on how you are querying, because generally that's not what you'd do and is inefficient.

So we have many groups of indices (technical-logs*, notifications*, incidents*, ...). We do a rollover daily so there is always date suffix. If we don't provide the date range, we are searching in indices with asterisk (e.g. technical-logs*). If we provide date range then we are trying to get all indices which are in given range (e.g. from technical-logs-20230101 to technical-logs-20230118 ). Now the question is if this is really necessary and if ES doesn't apply such filter automatically when we filter by @timestamp.

[warkolm (Mark Walkom)]

@bilak

the question was more regarding this topic

Ah ok. I don't know to be honest. If you follow that through to Migrate to data stream API | Elasticsearch Guide [7.17] | Elastic it says;

If successful, the request removes the alias and creates a data stream with the same name. The alias’s indices become hidden backing indices for the stream. The alias’s write index becomes the stream’s write index.

So my guess it doesn't do much other than some alias switching/renaming.

@bilak

Now the question is if this is really necessary and if ES doesn't apply such filter automatically when we filter by @timestamp.

Elasticsearch handles this transparently for you, so I wouldn't worry too much.