I am upgrading an existing cluster from ES 7.7.1 (lucene 8.5.1) to ES 7.12.0 (lucene 8.8.0) and have hit a massive performance regression in basic match queries utilizing minimum should match. I believe this is due to LUCENE-9346 which should have improved performance.
My queries are simple match queries against a single field, OR operator, and a minimum should match value usually between 20-50%. The search text ranges from 10 to 50 tokens after analysis. In ES 7.12.0 these queries are running between 4-20x slower than they did in ES 7.7.1. If I remove the minimum should match parameter they run ~2-3x FASTER than in ES 7.7.1 without the minimum should match value but still about 3x slower than ES 7.7.1 with minimum should match.
Here sample took times running the same query against the different es version with and without min should match parameter. Same cluster + index specs, same exact data, fully optimized single segment index.
ES 7.7.1 with min should match: 50ms
ES 7.12.0 with min should match: 1022ms
ES 7.7.1 without min should match: 350ms
ES 7.12.0 without min should match: 167ms
In general looks like WAND scoring has been improved but the min should match support introduced in LUCENE-9346 might have made things significantly worse.
Any ideas? I have hot threads output and it pretty clearly shows the time is spent in WANDScorer and DisiPriorityQueue:
app//org.apache.lucene.search.DisiPriorityQueue.upHeap(DisiPriorityQueue.java:135) app//org.apache.lucene.search.DisiPriorityQueue.add(DisiPriorityQueue.java:103) app//org.apache.lucene.search.WANDScorer.pushBackLeads(WANDScorer.java:316) app//org.apache.lucene.search.WANDScorer.doNextCompetitiveCandidate(WANDScorer.java:456) app//org.apache.lucene.search.WANDScorer.access$400(WANDScorer.java:46) app//org.apache.lucene.search.WANDScorer$1.advance(WANDScorer.java:267) app//org.apache.lucene.search.WANDScorer$1.nextDoc(WANDScorer.java:244) app//org.apache.lucene.search.Weight$DefaultBulkScorer.scoreRange(Weight.java:262) app//org.apache.lucene.search.Weight$DefaultBulkScorer.score(Weight.java:239) app//org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:45) app//org.apache.lucene.search.BulkScorer.score(BulkScorer.java:39)
12 - c5d.4xlarge data nodes with 16g heaps
~440M docs, 36 shards, 1 replica, index fully optimized
Single field indexed / searched with index_options freqs