Slow script query

Elastic Stack Elasticsearch
1

We have a 4 node cluster. Searching On a index with 40k records is taking upto 3 secs. It used to take less than a second like 400ms but now in last couple of weeks, it started slowing down. We dont see much of GC in the logs

We are on ES 1.4.2. 2 Nodes are allocated 31 Gb of 64 GB heap and other 2 nodes are allocated 15GB of 32 GB heap

us_zip_codes/_search
{
  "query": {
    "filtered": {
      "query": {
        "match_all": {}
      },
      "filter": {
        "script": {
          "script": "doc['zip_code'].values.findIndexOf{ srcVal -> param1.any{ it =~ /^(?i).*\\Q${srcVal}\\E.*/}} !=-1",
          "lang": "groovy",
          "params": {
            "param1": [
              "94555-2750"
            ]
          }
        }
      }
    }
  }
}

us_zip_codes has 5 digit zip_codes and we want to find a match when have a 9 digit code. I am using above script. How do i troubleshot this issue?

Also, is there a better way to re-write that query. Any help is greatly appreciated

2

We restarted the cluster last night and query performance is back to normal. Can someone give me some idea about what resource ES servers could have been holding to cause the slowness.

Note that, before cluster restart I cleared the cache multiple times and but that didn't help.

3

Hey,

in Elasticsearch 2.3 there were memory leaks with groovy scripts. You may have hit one in that pretty ancient version as well. This is just a wild guess, if you need to debug this further, you should analyze a heap dump probably.

--Alex

4

Thanks Alex. We are planning to upgrade to 2.4.4 pretty soon. But looks like that may not help either based on this discussion