Unexplained Speed Improvement in Third and Subsequent Terms Aggregations

Elastic Stack Elasticsearch
1

Hi @Mark_Harwood, I implemented @dadoonet's suggestion (eager global ordinals, thank you for this) in Slow terms aggregation speed on ~130M documents and was able to decrease the first load to ~2s from ~5s.

However, using random queries in the form:

{  
    "query":{
    	"bool": {
    		"must": {
    			"match_phrase":{  
	                "_all":"randomtext"
	            }
    		}
    	}
    },
    "_source":[],
    "size": 0,
    "aggregations":{  
        "suggestions":{  
            "terms":{  
                "field":"Consignee.Name.untouched",
                "size":100
            }
        }
    }
}

I get the following query time pattern, regardless of queries used:
1st search: ~2s
2nd search: ~2s
3rd search: < 1s
4th search: < 1s
...
Xth search: < 1s

My questions are:

  1. Why does it speed up on the third query (I suspect this is from some kind of OS or ES level caching in the background)?
  2. Is there some way I can also preemptively do this?

This question is just out of curiosity, I am perfectly happy with the current performance.

2

Round-robining over replicas?

1 Like
3

Which question are you answering? Also, can you please elaborate? Thank you in advance. Also, it seems like it's not specifically on the third query. I retried the setup but this time, it speeds up on the fourth query.

4

Sorry, question 1. If your client is balancing requests across replicas any caching on your primary shard will not benefit a follow-up query routed to your replica.
To send each user back to the same replica each time and increase the chances of hitting a warm cache use their sessionID as a routing preference
Looking at the docs there it seems randomization and not round-robining is the default replica selection policy which may explain some of the inconsistencies.

5

If I understand correctly, then the first search will always be longer than the others? The second search is not guaranteed to be the same node as the first search's and the search improves for the third and all subsequent searches because I have three nodes (which means the possibility of hitting a warm cache by the third try is 83%, 100% on 4th)?

That makes a lot of sense, thank you. For question 2 then, is there a way to 'warm up' the caches before the user searches?

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.