Is it possible to "warm up" indexes with low volume?

Elastic Stack Elasticsearch
1

I have an index with 9M documents, 1 shard and 1 replica that receives traffic very sporadically (less than 5 requests per hour). The ingestion rate of this index is less than 1/s

A recurring pattern is that query response times spikes to 500ms–1s or higher immediately after the index is accessed after a long time. However, when the index becomes "hot" (i.e., receives more frequent requests), response times decrease significantly to the expected range for an index of this size, typically around 10–20ms.

Is there any way to configure Elasticsearch to “always keep an index hot”?

In case it matters, I have full control of the clients and know exactly what the query patterns are

2

What about running your query from a crontab for example?

If you are running on cloud (Gold) or with a commercial license you might want to run this frequent query from Watcher. Using Search input | Elastic Docs for example.

1 Like
3

Thank you for your response @dadoonet

This is an approach I’ve thought about but haven’t tested. Is this how it is usually solved?

My understanding has been that caches are specific to each query and value. Is this understanding wrong?

Would running a scheduled query with a fixed value be beneficial in this case?

In other words, are caches maintained at the index/field level, or are they specific to the combination of index, field, and value?

4

Also, one follow up question if you don’t mind @dadoonet

While designing this sort of “Warming”, should I run the query patterns that we have for these indexes or would a single match all do the trick?

5

You should run a typical query, multiple times. With the right field names.
If you are using a filter by date, I'd suggest to use non static values but something like to: now/d+1d from now/d-30d for example.

Don't forget to add yours aggs, sort... if any.

6

Have you set index.refresh_interval on this index? If so, how is it configured?