# Omit indices when searching on multiple indices

**URL:** <https://discuss.elastic.co/t/omit-indices-when-searching-on-multiple-indices/323866>\
**Category:** Elasticsearch\
**Created:** [January 24, 2023, 7:31pm UTC](https://discuss.elastic.co/t/omit-indices-when-searching-on-multiple-indices/323866 "2023-01-24T19:31:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [January 24, 2023, 7:31pm UTC](https://discuss.elastic.co/t/omit-indices-when-searching-on-multiple-indices/323866/1 "2023-01-24T19:31:59Z")

</div>

A question and maybe a feature request. We have 100 indices for filebeat with this pattern:

> filebeat-{CLUSTER\_NAME}-{NAMESPACE}-{DATE}

.

Documents are:

```auto
{"@timestamp" : "..." , "cluster" : "prod", "namespace" : "kube-system", ... "message": "hello world"}

```

I am wondering, when we search against all filebeat indices, for a particular "time range" and "cluster", **if elasticsearch is smart enough to select first potential good indices?**

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [January 25, 2023, 3:07am UTC](https://discuss.elastic.co/t/omit-indices-when-searching-on-multiple-indices/323866/2 "2023-01-25T03:07:22Z")

</div>

Hi @ebuildy

What version are you using?

The short answer is there are "Smarts" built into elasticsearch to "Prefetch/Limit" the applicable indices based on timestamps with respect to the time range of your search **IF** you are doing normal times series data ingestion with rollover/daily etc ILM etc and not reopening and writing to them etc.. etc..

Elastic **will not know** about that CLUSTER or NAMESPACE name in the index name and will not pre-filter that UNLESS you create a data view or something to limit the search upfront

So your answer is yes and no....

Others may have more details... but that is my top-level understanding

---

<div class="post-metadata">

**Author:** ![ebuildy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ebuildy/32/6070_2.png) [@ebuildy](https://discuss.elastic.co/u/ebuildy)\
**Post date:** [January 25, 2023, 1:32pm UTC](https://discuss.elastic.co/t/omit-indices-when-searching-on-multiple-indices/323866/3 "2023-01-25T13:32:23Z")

</div>

Very interesting,

I know this is the 1st optimisation step of some DBs: "dont open file / resource if you dont need it".

I am a big fan of Apache Spark and all data stuff (parquet, data lake etc...), they do something called **"Partition pruning"** to work only on good files, elasticsearch could implement this concept.

If I do the analogy with Parquet file format, elasticsearch could save for each index the min and max value for time fields, terms values for string fields (with a limit), and pre-filter indices before doing the search.

* * *

So as a good advice, this is better to group all documents by date indices (less indices but bigger): "filebeat-YYYY-MM-DD" than indices like (more indices but smaller) "filebeat-{PRODUCT}-YYYY-WEEK"

Thanks you,

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 22, 2023, 2:21pm UTC](https://discuss.elastic.co/t/omit-indices-when-searching-on-multiple-indices/323866/5 "2023-02-22T14:21:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.