I have 1000+ index
myindex-<date>-00000x ( this is all rollover using ILM)
in my REST calll should I use
myindex-* or use latest index by it's name like "
most everytime I retrive data using @timestamp because this is timeseries data
"@timestamp" > now() - interval X hour
when I use myindex-* does it scan all the index or just latest one?
Elasticsearch will use Field capabilities API | Elasticsearch Guide [8.7] | Elastic to exclude indices that may not match your query, so you can use wildcards.
but sometime when I use my-index-* it timesout.
but exactly on same time if I use myindex-toda_date-xxxx it works.
Are you using a date range filter in you query?
yes date range. this is metricbeat index and using @timestamp, all the data are timeseries
The query should pre-filter the indices based on the time range when using a data view or wildcard.
Use the search profiler to see what is going on.
Example when I put this is the profiler
I see that it only searches the relevant indices ... take a look what yours is doing
Wow I never used this search profiler.
in my case it is not doing optimal it is even going some random index which are few week old. and I used exact query that you had.
for obvious reason I can't copy-paste screen shot of it. but it is looking in to total 7 index on "query Profile" window.
my metricbeat index is xyz-7.17.1 but it is looking in to four of the old index xyz-7.12.0
hmm. but if I select index as latest one it profiler shows only one being used, as obvious.
it seems like behavior are different on manually created rollover index (via pattern/ilm) then data stream.
I have another system which I updated to 8.5.3 and has metricbeat data handled by data stream is doing exactly what expect to do.
i.e if i use date 2023-04-10 then it is only looking in to that day index.
if I use date 2023-01-01 then it is only looking in to that day index.
I guess I will have to live with old way untill I can upgrade them.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.