Slow response times query first couple of times

Elastic Stack Elasticsearch
1

We have a service which is using ES sporadically, (once every few hours). We noted that the first couple of times we query ES takes a lot longer than subsequent calls. For example if we perform the following query:

{
    "query": {
        "bool": {
            "must": [
                
                    {"term": {
                        "organization.id": 1234
                    }},
                    {"match": {
                        "name": "BEST HITS"
                    }}
                
            ]
        }
    }
}

the first call can take up to 5 seconds. Subsequently calls decrease rapidly (2 seconds, 1100 ms, 100 ms). We have done some experimenting with the property index.search.idle.after and are now testing changing index.refresh_interval but I'm wondering whether we are on the right track. All help will be appreciated.

2

If you are querying ES rarely, but actively indexing, Elasticsearch moves shards to "idle" state for search which means that search indices are not actively built. And once after some time you hit a search request, for all these new indexed data new search indices will be built, which may take some time. This is the best optimization for indexing speed.

If you want your searches to be always faster even after some period of inactivity, you can set up an explicit "index.refresh_interval", which ensures that new search indices are built after this interval.

Another factor that comes into play for repeated queries are caching. If you run the same query multiple times, its results may be cached, and subsequent calls for the same query can return much faster, as they are returned from cache. This is provided that your index doesn't change.

3

Thanks for the quick response and your confirmation that we are on the right track. We are today doing some more tests and I will put the results of them here.