Use data view or direct index

Elastic Stack Elasticsearch
1

I have 1000+ index

myindex-<date>-00000x ( this is all rollover using ILM)

in my REST calll should I use
myindex-* or use latest index by it's name like "myindex-<date>-00000x"

most everytime I retrive data using @timestamp because this is timeseries data
"@timestamp" > now() - interval X hour

when I use myindex-* does it scan all the index or just latest one?

2

Elasticsearch will use Field capabilities API | Elasticsearch Guide [8.7] | Elastic to exclude indices that may not match your query, so you can use wildcards.

3

Hi Mark,
but sometime when I use my-index-* it timesout.
but exactly on same time if I use myindex-toda_date-xxxx it works.

4

Are you using a date range filter in you query?

5

yes date range. this is metricbeat index and using @timestamp, all the data are timeseries

6

The query should pre-filter the indices based on the time range when using a data view or wildcard.

Use the search profiler to see what is going on.

Example when I put this is the profiler

logs-*

{
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2023-04-19T13:51:20.382Z",
              "lte": "2023-04-19T14:06:20.382Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

I see that it only searches the relevant indices ... take a look what yours is doing

Screenshot 2023-04-19 at 7.09.52 AM
Screenshot 2023-04-19 at 7.09.52 AM1920×1019 124 KB

7

Wow I never used this search profiler.

in my case it is not doing optimal it is even going some random index which are few week old. and I used exact query that you had.
for obvious reason I can't copy-paste screen shot of it. but it is looking in to total 7 index on "query Profile" window.

my metricbeat index is xyz-7.17.1 but it is looking in to four of the old index xyz-7.12.0

hmm. but if I select index as latest one it profiler shows only one being used, as obvious.

8

Hi Stephen,
it seems like behavior are different on manually created rollover index (via pattern/ilm) then data stream.

I have another system which I updated to 8.5.3 and has metricbeat data handled by data stream is doing exactly what expect to do.

i.e if i use date 2023-04-10 then it is only looking in to that day index.
if I use date 2023-01-01 then it is only looking in to that day index.

I guess I will have to live with old way untill I can upgrade them.