Wildcard queries slow since ES 5.x

[nesretep]

Hi there
In the past, to my knowledge untill ES 6, queries - especially wildcard queries - were in general blisteringly fast. Since then, performance has degraded dramatically, with small improvements up through ES 7 and now 8.
One thing which we suspect, is the absense of the _all field, which may have been important for fast queries. The new "wildcard" type helped a bit, bur we are still lightyears from the original performance of ES 5. Is there any possible way of achieving better performance for wildcard queries (and we are talking factor 100 or 1000's), in ES8?

[Christian_Dahlqvist]

Wildcard queries have always been one of the most expensive queries in Elasticsearch as a lot of terms need to be scanned and compared to the wildcard. I am not aware of any general dramatic decrease in performance in Elasticsearch 6.x.

In early versions of Elasticsearch doc_values were not used by default and terms were kept on the heap, which led to high heap usage. I guess this might have helped speed wildcard queries up, especially if you have slow storage. If you wanted to test this I suspect you can still control this through the field mapping if you want to test it.

If you are running wildcard queries againt all fields and not specific ones it is possible that the use of the _all field may have helped. You can however get largely the same effect in recent versions using the copy_to functionality. You should therefore be able to test this as well in a recent version to see if it makes a difference.

Those are the things I can think of that could affect this, but there may be others. It would help if you performed benchmarks on the old and new versions and shared data, mappings and queries so we can tell whether this is the case or not.

If you do want to run wildcard queries against a lot of fields in the latest version of Elasticsearch I would test creating a copy_to field mapped with the wildcard field type and run try running queries against this.