# Would the search performance be very slow if a data stream consists of too many or too large cold backing indices? **URL:** \ **Category:** Elasticsearch\ **Tags:** datastreams\ **Created:** [January 14, 2022, 7:46am UTC](https://discuss.elastic.co/t/would-the-search-performance-be-very-slow-if-a-data-stream-consists-of-too-many-or-too-large-cold-backing-indices/294376 "2022-01-14T07:46:26Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Morriaty](https://avatars.discourse-cdn.com/v4/letter/m/8e8cbc/32.png) [@Morriaty](https://discuss.elastic.co/u/Morriaty)\ **Post date:** [January 14, 2022, 7:46am UTC](https://discuss.elastic.co/t/would-the-search-performance-be-very-slow-if-a-data-stream-consists-of-too-many-or-too-large-cold-backing-indices/294376/1 "2022-01-14T07:46:26Z")
Hi, I'm moving from es 6.x to 7.x, find the new feature `data stream` and would like to have a try. But I'm a bit worried of the search performace for the docs explains `When you submit a read request to a data stream, the stream routes the request to all its backing indices.` [doc source](https://www.elastic.co/guide/en/elasticsearch/reference/current/data-streams.html#data-stream-read-requests) Say we have a data stream alias as `audit_log`, consisting of hundreds of backing indices: - `.ds.audit_log_2022.01_000001` HOT PHASE - `.ds.audit_log_2021.12_000002` COLD PHASE - `.ds.audit_log_2021.12_000001` COLD PHASE ..... - `.ds.audit_log_2021.01_000100` COLD PHASE If we do search requests: ```json GET audit_log/_search { "query": { "bool": { "filter": { "range": { "@timestamp": { "gte": "now-1M/d" } } }, "must": [ { "match": { "auditor": "tom" } } ] } } } ``` 1. Can this search request only route to the several backing indices cause of the `@timestamp range clause`? 2. If the above answer is NO. Is there any other alternative features could help us to route to indices according to `date fileds` by default. ---
**Author:** ![Morriaty](https://avatars.discourse-cdn.com/v4/letter/m/8e8cbc/32.png) [@Morriaty](https://discuss.elastic.co/u/Morriaty)\ **Post date:** [January 17, 2022, 4:17am UTC](https://discuss.elastic.co/t/would-the-search-performance-be-very-slow-if-a-data-stream-consists-of-too-many-or-too-large-cold-backing-indices/294376/2 "2022-01-17T04:17:15Z")
Found a question alike: [Elasticsearch search query on hot and warm nodes - Stack Overflow](https://stackoverflow.com/questions/62171313/elasticsearch-search-query-on-hot-and-warm-nodes) Regarding to data streams, the problems turn to be: 1. set aliases `ds_search_recent` and `ds_search_all` to the data stream 2. remove alias `ds_search_recent` when entering into warm(cold) phases But currently seems there is no automatic way to do these ---
**Author:** ![Morriaty](https://avatars.discourse-cdn.com/v4/letter/m/8e8cbc/32.png) [@Morriaty](https://discuss.elastic.co/u/Morriaty)\ **Post date:** [January 17, 2022, 6:59am UTC](https://discuss.elastic.co/t/would-the-search-performance-be-very-slow-if-a-data-stream-consists-of-too-many-or-too-large-cold-backing-indices/294376/3 "2022-01-17T06:59:14Z")
A related github issue [Add ILM action to add/remove aliases · Issue #47881 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/47881) Bad news it is still open since 2019 ---
**Author:** ![Morriaty](https://avatars.discourse-cdn.com/v4/letter/m/8e8cbc/32.png) [@Morriaty](https://discuss.elastic.co/u/Morriaty)\ **Post date:** [January 17, 2022, 7:41am UTC](https://discuss.elastic.co/t/would-the-search-performance-be-very-slow-if-a-data-stream-consists-of-too-many-or-too-large-cold-backing-indices/294376/4 "2022-01-17T07:41:18Z")
Another related PR [Use @timestamp field to route documents to a backing index of a data stream by martijnvg · Pull Request #82079 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/pull/82079) It seems to be a new feature of es 8.1, which is not released yet. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 17, 2022, 10:13am UTC](https://discuss.elastic.co/t/would-the-search-performance-be-very-slow-if-a-data-stream-consists-of-too-many-or-too-large-cold-backing-indices/294376/5 "2022-01-17T10:13:49Z")
> 1. Can this search request only route to the several backing indices cause of the `@timestamp range clause` ? Technically no, but in practice ES achieves this goal anyway: it routes the search to every backing index but the ones that don't match the timestamp range get optimised into a `MatchNoDocsQuery` which obviously hits no data and takes no time to execute. ---
**Author:** ![Morriaty](https://avatars.discourse-cdn.com/v4/letter/m/8e8cbc/32.png) [@Morriaty](https://discuss.elastic.co/u/Morriaty)\ **Post date:** [January 18, 2022, 6:54am UTC](https://discuss.elastic.co/t/would-the-search-performance-be-very-slow-if-a-data-stream-consists-of-too-many-or-too-large-cold-backing-indices/294376/6 "2022-01-18T06:54:36Z")
> [@DavidTurner](#): > > it routes the search to every backing index but the ones that don't match the timestamp range get optimised into a `MatchNoDocsQuery` Thank you David. I am still curious that is this a feature of backing indices or of filter cache? I mean would it be slow at the first search query and slow when not hit time span filter cache? For that common time series search cases always changed time span frequently. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 18, 2022, 8:16am UTC](https://discuss.elastic.co/t/would-the-search-performance-be-very-slow-if-a-data-stream-consists-of-too-many-or-too-large-cold-backing-indices/294376/7 "2022-01-18T08:16:45Z")
This happens when rewriting the query into its optimised form, long before the filter cache gets involved. It involves comparing two `long` timestamp values, which is almost instantaneous and doesn't involve any caching. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 15, 2022, 8:16am UTC](https://discuss.elastic.co/t/would-the-search-performance-be-very-slow-if-a-data-stream-consists-of-too-many-or-too-large-cold-backing-indices/294376/8 "2022-02-15T08:16:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.