Wildcarded query make elasticsearch cluster unresponsive

Elastic Stack Elasticsearch
1

I run the query below on a large elastic search cluster. The cluster bcomes unresponsive

{
  "size": 10000,
  "query": {
    "bool": {
      "must": [
        {
          "regexp": {
            "message": {
              "value": ".*exception.*"
            }
          }
        },
        {
          "bool": {
            "should": [
              {
                "term": {
                  "beat.hostname": "ip-xxx-xx-xx-xx"
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "lt": 1518459660000,
              "format": "epoch_millis",
              "gte": 1518459600000
            }
          }
        }
      ]
    }
  }
}

When I remove the wildcarded .*exception.* and replace it with any non wildcarded string like xyz it returns fast. Though the query uses a wildcarded expression, it also looks for a small time range and a specific host. I would think this is a very simple query. Any reason why elasticsearch server can't handle this query? The cluster has 10 nodes and 20 TB of data.

2

Probably the reason we are saying that wildcards are slow: Wildcard query | Elasticsearch Guide [8.11] | Elastic

Note that this query can be slow, as it needs to iterate over many terms. In order to prevent extremely slow wildcard queries, a wildcard term should not start with one of the wildcards * or ?.

3

Thanks for the response. I read that document before. If you look at the query I posted, I restrict the query based on the timestamp range and the hostname term. I handpicked these values so no documents are matched.
It still causes the cluster to become unresponsive.

So is it fair to assume that elasticsearch does not optimize the query and randomly picks what field to query first?

4

Put the other queries inside a filter clause instead of the must clause.
Or at least, put the worse query at the end.

5

Thanks for the quick response. I will give that a try. Is there a way I can verify that filter happens first, maybe through a explain plan?

6

Profile API might give some clues.

I think something like this is coming but I don't remember from the top of my head.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.