Werid err: org..CircuitBreakingException: [script] Too many dynamic script compilations

[stanats]

we are getting weird error : org.elasticsearch.common.breaker.CircuitBreakingException: [script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting

[o.e.a.s.TransportSearchAction] [yyvrm-z] [testindex][2], node[yyvrm-zXSKGpvwChY5dm_Q], [P], s[STARTED], a[id
=M3POIs_SSqigKhUlwKZXYA]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[], indicesOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expa
nd_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_aliases=false, ignore_throttled=true], types=[], rout
ing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=15, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClust
erAlias=null, getOrCreateAbsoluteStartMillis=-1, source={"size":1,"query":{"match_all":{"boost":1.0}},"script_fields":{"myscript":{"script":{"source":"java.lang.Math.class.forName(\"j
ava.lang.System\").getProperties()","lang":"painless"},"ignore_failure":false}}}}] lastShard [true]

We dont have "myscript" in our ES queries  , is this something ES is doing internally ?
Please help

[polyfractal (Zachary Tong)]

@stanats

{"size":1,"query":{"match_all":{"boost":1.0}},"script_fields":{"myscript":{"script":{"source":"java.lang.Math.class.forName("j ava.lang.System").getProperties()","lang":"painless"}

Looks like someone is attempting to use an old Groovy security exploit on your cluster. See Disable dynamic Groovy scripting by marking Groovy as not sandboxed · Issue #9655 · elastic/elasticsearch · GitHub for more details.

Basically, that script was able to escalate privileges in older version of Elasticsearch that used Groovy (~4 years ago, in the 1.x series).

It's harmless today as the security vulnerability was A) patched and B) we don't use Groovy anymore (one of the main reasons we created Painless was to ensure it could be locked down from a security perspective). With that said, you should probably not allow users to inject arbitrary script fields or other portions of the query into the search request, just a matter of general safety.

[stanats]

Thank you so much for the info.
Can you please advice how to stop injecting arbitrary scripts /query part of search request ?

Thanks
Stan

[polyfractal (Zachary Tong)]

That's mostly a front-end/UI operation. E.g. you need to either translate a user's request into the query DSL, or sanitize the input somehow. It's not the same as a SQL injection, but you can consider the manner of validation the same: don't ever let user input go directly into a SQL query (or an Elasticsearch query) without some kind of validation/sanitization.