I am working on tuning a search query pipeline in Elasticsearch and running into a performance bottleneck when executing high-frequency wildcard and regular expression searches against a rapidly growing index.
Current Setup:
Elasticsearch version: 8.x cluster
Index size: Multi-gigabyte log index with text and keyword fields
Query pattern: Using wildcard and regexp queries extensively for operational filtering.
The Bottleneck: As document volume scales, these queries are bypassing standard term dictionary lookups, causing higher CPU usage and slower response times across concurrent user requests.
Are others utilizing index_options adjustments, runtime fields, or alternative mapping strategies (like match_phrase_prefix or edge n-grams) to speed up pattern matching without degrading cluster stability?
Would appreciate any recommendations or indexing best practices for improving wildcard search performance. Thanks!
In 9.x we've done some work, as has the Lucene community, to speed up regexes but it probably won't be sufficient to solve your problems long term
Are others utilizing index_options adjustments, runtime fields, or alternative mapping strategies (like match_phrase_prefix or edge n-grams) to speed up pattern matching without degrading cluster stability?
i'd be curious what other folks think here but I doubt you'll get a solution generally speaking; regexes are just inherently slow. Targeting specific types of queries you get in volume might help a good bit though so you mention prefix queries for instance and if you can afford to index the data with index_prefixesspecific options as a for instance and push your query patterns that are prefix in nature into that model then you'll see a big improvement.
But if you don't have a ton of control of the regexes being used my head goes to two places either you can try to structure and at query time filter on your data more so or you can try to scale up by adding more replicas which will definitely at some point still hit a limit.
As far as structuring your data you can always come back in and modify your mappings, add fields, and use filters to reduce the total explored set of data as you learn more about your logs or different types of logs.
But also specifically for logs data ideally you'd be able to temporally filter the data for most use cases to say by default a few days (and maybe a few default other filters) and by that mechanism control the latency of typical queries indefinitely. You could maybe even roll over to new indices occasionally and allocate more hardware to more recently indexed log data and less hardware to more archival data that you could search more slowly. I'm making some big assumptions here so happy to talk through any additional thoughts or questions particularly if you want to share mappings or any additional details.
Can you give some example of the messages / text and the regex queries.
I ask because what I have found is that sometime users are using regex because the OOTB tokenizer don't give the results they want, especially on machine generated logs... sometimes some fairly minor changes to the tokenizer then allows much more use of full text search and less reliance of regex.
Can you provide some samples of the data, regex and expected results?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.