Use data view or direct index

I have 1000+ index

myindex-<date>-00000x ( this is all rollover using ILM)

in my REST calll should I use
myindex-* or use latest index by it's name like "myindex-<date>-00000x"

most everytime I retrive data using @timestamp because this is timeseries data
"@timestamp" > now() - interval X hour

when I use myindex-* does it scan all the index or just latest one?

Elasticsearch will use Field capabilities API | Elasticsearch Guide [8.7] | Elastic to exclude indices that may not match your query, so you can use wildcards.

Hi Mark,
but sometime when I use my-index-* it timesout.
but exactly on same time if I use myindex-toda_date-xxxx it works.

Are you using a date range filter in you query?

yes date range. this is metricbeat index and using @timestamp, all the data are timeseries

The query should pre-filter the indices based on the time range when using a data view or wildcard.

Use the search profiler to see what is going on.

Example when I put this is the profiler


  "query": {
    "bool": {
      "must": [],
      "filter": [
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2023-04-19T13:51:20.382Z",
              "lte": "2023-04-19T14:06:20.382Z"
      "should": [],
      "must_not": []

I see that it only searches the relevant indices ... take a look what yours is doing

1 Like

Wow I never used this search profiler.

in my case it is not doing optimal it is even going some random index which are few week old. and I used exact query that you had.
for obvious reason I can't copy-paste screen shot of it. but it is looking in to total 7 index on "query Profile" window.

my metricbeat index is xyz-7.17.1 but it is looking in to four of the old index xyz-7.12.0

hmm. but if I select index as latest one it profiler shows only one being used, as obvious.

Hi Stephen,
it seems like behavior are different on manually created rollover index (via pattern/ilm) then data stream.

I have another system which I updated to 8.5.3 and has metricbeat data handled by data stream is doing exactly what expect to do.

i.e if i use date 2023-04-10 then it is only looking in to that day index.
if I use date 2023-01-01 then it is only looking in to that day index.

I guess I will have to live with old way untill I can upgrade them.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.