I have a deployment of ~30 node ES Cluster, where the index size density is roughly 2 TB per node.
We have a use-case of some ES node regularly crashing when our end-user triggers wildcard searches. (ES node goes OOM)
While we are putting some guardrails in UI to stop user from doing bad searches(for ex: if somebody only searches with *), my question is that is there a way to upfront predict a bad search request & terminate it, instead of crashing the server,
The memory usage based on heapdump seems to be mainly around the # of possible expansions, these wildcard searches generates.
For ex : ru* generating 1M+ expansions.
I came across "Multi-Term Query Rewrite" option below. It looks compelling enough to try, i.e. try to limit the # of possible expansions of a wildcard search upfront, instead of allowing it to expand in an unlimited fashion.
https://www.elastic.co/guide/en/elasticsearch/reference/2.4/query-dsl-multi-term-rewrite.html
Has anyone used this option for a similar use-case ?
Will it help drastically improving the reliability of my ES nodes going OOM ?
Note : We use span_multi queries for the business use-case we support. (ES Version 2.4)