Too many dynamic script compilations within one minute

Elastic Stack Elasticsearch
1

I am facing "max_compilations_per_minute" error.

TransportError(500, '{"error":{"root_cause":[{"type":"circuit_breaking_exception","reason":"[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting","bytes_wanted":0,"bytes_limit":0},

This is mainly happening due to the painless script we are using in in search queries registered as percolator.

Since we are using regex in our queries Indexed or parameter script is not an option.

Re-indexing of data is not an option for us.
How can we avoid this circuit break exception happening mainly due to "max_compilations_per_minute".

Alternative or worst solution:-
Is there any max limit for setting "max_compilations_per_minute" recommended

1 Like
2

I am not following what you are saying here. Mind giving an example of your search with script in which indexed and parameter can't be used?

3

Here is one of sample query.

   {
   "query": {
      "bool": {
         "must": [
            {
               "query_string": {
                  "query": "a*",
                  "quote_analyzer": "standard",
                  "allow_leading_wildcard": "true",
                  "analyze_wildcard": "true",
                  "default_operator": "AND",
                  "phrase_slop": 1
               }
            },
            {
               "script": {
                  "script": {
                     "source": "Pattern p =  /\\s?a\\S*?/i;   p.matcher(params._source.summary ?: '').find();",
                     "lang": "painless"
                  }
               }
            }
         ]
      }
   }
}
4

Instead of using the native regex syntax, use Pattern. compile which accepts the pattern as a string that could come from a parameter. For instance, Pattern. compile(params.regex, "i"). Then you can use this as a stored script.

5

Did you mean something like this?

{
  "query": {
    "bool": {
      "must": [
        {
          "bool": {
            "must": [
              {
                "query_string": {
                  "query": "a*",
                  "quote_analyzer": "standard",
                  "allow_leading_wildcard": "true",
                  "analyze_wildcard": "true",
                  "default_operator": "AND",
                  "phrase_slop": 1
                }
              },
              {
                "script": {
                  "script": {
                    "lang": "painless",
                    "source": """Pattern p = Pattern.compile(params.param1) ;p.matcher(doc['headline'].value ?: '').find();""",
                    "params": {
                      "param1": "/\\s?a\\S*?/i"
                    }
                  }
                }
              }
            ]
          }
        }
      ]
    }
  }
}
6

by the way, can you share the official documentation about this Pattern. compile(params.regex, "i")

Documentation says:

Regular expression constants are directly supported. To ensure fast performance, this is the only mechanism for creating patterns. Regular expressions are always constants and compiled efficiently a single time.

7

It seems Pattern. compile is not supported in painless script.

https://www.elastic.co/guide/en/elasticsearch/painless/current/painless-syntax.html#patterns
Or I am missing something.
I am using version 5.6

8

Sorry for dropping the ball here. Indeed Pattern.compile is not supported, which makes it technically impossible to create a stored painless index with a variable Pattern, like the way I have described.

There is a performance concern in blocking the Pattern.compile, since it makes the regex compilation dynamic. So I can think of a couple of alternatives:

  1. If you have a known and small number of patterns, then you can create a stored script that knows all of them (and selects on using a variable), or maybe many stored scripts, which one for each pattern.
  2. If you don't have a known set of patterns or the list is just too big, then consider implementing your script as a native script and use some internal LRU caching when dynamically compiling the pattern.
9

we are left with native script since we have a huge set of wildcard queries(obviously now it will be regex script).

So, you suggest that in our case use native script and increase the LRU cache.

what if we increase script.cache.max_size and script.max_compilations_per_minute - will that help if we have more than 100K (increases but not in huge numbers) of dynamic scripts?

10

You can certainly increase script.max_compilations_per_minute so Elasticsearch would allow you running hundreds of thousands of scripts per minutes but that also means a potential huge CPU spike that will hurt performance.

I suggest that you benchmark and verify if you can live with the high CPU usage (if any at all). If that does not works for you, then I suggest that you follow the native script with LRU strategy.

2 Likes
11

@thiago Thanks for your input.

12

Stored script:

POST _scripts/536672d3cface3ddbb63666bf1b6030f
{
  "script": {
      "lang": "painless",
      "source": "Pattern p = /\\s?a\\S*?/i; p.matcher(params._source.summary ?: '').find();"
   }
}

I am using the ID of the stored script in the query like

{
   "query": {
      "bool": {
         "must": [
            {
               "bool": {
                  "must": [
                     {
                        "query_string": {
                           "query": "a*",
                           "quote_analyzer": "standard",
                           "allow_leading_wildcard": "true",
                           "analyze_wildcard": "true",
                           "default_operator": "AND",
                           "phrase_slop": 1
                        }
                     },
                     {
                        "script": {
                           "script": {
                              "id": "536672d3cface3ddbb63666bf1b6030f"
                           }
                        }
                     }
                  ]
               }
            }
         ]
      }
   }
}

All these queries are stored in a percolator. If a document comes, it will percolated. That . time it fails with [script] Too many dynamic script compilations within one minute, max

any solution?

Even if the scripts are stored in the index, will it execute each and every time?

I hope these settings are for cluster wide, I thought of making two or more indexes and do the percolation so that per minute compilation will for each and every index. Since documentation does not say anything about caching much like where it resides how Elasticsearch is using that, I am little confused here!

1 Like
13

You can increase script.max_compilations_per_minute. Here is an example which increases the parameter to 100

PUT _cluster/settings
{
    "transient" : {
        "script.max_compilations_per_minute" : 100
    }
}

Also, If you are using hard-coding values try to use named parameters instead. Scripts containing hard-coding values have to be recompiled every time the values changes. For more details see https://www.elastic.co/guide/en/elasticsearch/reference/5.5/modules-scripting-using.html

14

The query you have presented should not be the source of the Too many dynamic script compilations within one minute... error that you are getting now. It is something else.

By using a script id like that the script is not dynamically compiled anymore (it is compiled only once, when it was registered using the _script API) so there is no reason to trigger the script.max_compilations_per_minute threshold.

15

Yeah, after you told that it is compiling and storing, it is obvious that when we search in percolator it should not make any issue.

These are the scenarios:

  1. Registration API: it will register 1000s of scripted query in percolator index. Sometimes this is failing saying that 5000/min is reached. Make sense because we are registering(indexing) and while indexing it will compile. So it might be crossing the limit of execution.
  2. If that is the case percolator has 100K of queries and all are compiled since we are using scripted query. So when we search in percolator it should not throw this error of per minute execution issue. Suppose if it is making this issue you are saying that it might be the issue in the script, not about the compilation limit, isn't it? Sometimes I have seen that if there is any issue in the script it will throw this same error and not only me lots of people complained about this before. Anyway wrong error message from Elasticsearch.
16

Indeed the error you are seeing should because you are hitting the limits when registering the scripts. In this case, if this registration flood happens on a low frequency, then it is fine to bump the limit to a very high number so you don't hit the limit when it happens.

Regarding your second comment, I am not aware of any issues that makes Elasticsearch throw the limit error while shadowing another underlying issue. If Yiu can provide me an example or a way to reproduce it would be great.

17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.