Strange error message. To many dynamic script

[iukea (jordan)]

Hello, guys, I am using the Palo Alto and Cisco ASA module in file beats and I suddenly started getting the following error message.

error.message [script] Too many dynamic script compilations within, max: [100/1m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting

Any recommendations? Also suddenly none of my visualizations work anymore.

[iukea (jordan)]

The logs from the kibana

message":"Failed to poll for work: [circuit_breaking_exception] [script] Too many dynamic script compilations within, max: [100/1m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting, with { bytes_wanted=0 & bytes_limit=0 & durability=\"TRANSIENT\" } :: {\"path\":\"/.kibana_task_manager/_search\",\"query\":{\"ignore_unavailable\":true},\"body\":\"{\\\"query\\\":{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"type\\\":\\\"task\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"bool\\\":{\\\"should\\\":[{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.status\\\":\\\"idle\\\"}},{\\\"range\\\":{\\\"task.runAt\\\":{\\\"lte\\\":\\\"now\\\"}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.status\\\":\\\"running\\\"}},{\\\"range\\\":{\\\"task.retryAt\\\":{\\\"lte\\\":\\\"now\\\"}}}]}}]}},{\\\"bool\\\":{\\\"should\\\":[{\\\"exists\\\":{\\\"field\\\":\\\"task.interval\\\"}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"maps_telemetry\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":3}}}]}},{\\\"bool\\\":{\\\"must\\\":[{\\\"term\\\":{\\\"task.taskType\\\":\\\"vis_telemetry\\\"}},{\\\"range\\\":{\\\"task.attempts\\\":{\\\"lt\\\":3}}}]}}]}}]}}]}},\\\"size\\\":10,\\\"sort\\\":{\\\"_script\\\":{\\\"type\\\":\\\"number\\\",\\\"order\\\":\\\"asc\\\",\\\"script\\\":{\\\"lang\\\":\\\"expression\\\",\\\"source\\\":\\\"doc['task.retryAt'].value || doc['task.runAt'].value\\\"}}},\\\"seq_no_primary_term\\\":true}\",\"statusCode\":500,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"circuit_breaking_exception\\\",\\\"reason\\\":\\\"[script] Too many dynamic script compilations within, max: [100/1m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting\\\",\\\"bytes_wanted\\\":0,\\\"bytes_limit\\\":0,\\\"durability\\\":\\\"TRANSIENT\\\"}],\\\"type\\\":\\\"search_phase_execution_exception\\\",\\\"reason\\\":\\\"all shards failed\\\",\\\"phase\\\":\\\"query\\\",\\\"grouped\\\":true,\\\"failed_shards\\\":[{\\\"shard\\\":0,\\\"index\\\":\\\".kibana_task_manager_1\\\",\\\"node\\\":\\\"NE2b-0XkQSCcAQprVycfoQ\\\",\\\"reason\\\":{\\\"type\\\":\\\"general_script_exception\\\",\\\"reason\\\":\\\"Failed to compile inline script [doc['task.retryAt'].value || doc['task.runAt'].value] using lang [expression]\\\",\\\"caused_by\\\":{\\\"type\\\":\\\"circuit_breaking_exception\\\",\\\"reason\\\":\\\"[script] Too many dynamic script compilations within, max: [100/1m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting\\\",\\\"bytes_wanted\\\":0,\\\"bytes_limit\\\":0,\\\"durability\\\":\\\"TRANSIENT\\\"}}}],\\\"caused_by\\\":{\\\"type\\\":\\\"circuit_breaking_exception\\\",\\\"reason\\\":\\\"[script] Too many dynamic script compilations within, max: [100/1m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting\\\",\\\"bytes_wanted\\\":0,\\\"bytes_limit\\\":0,\\\"durability\\\":\\\"TRANSIENT\\\"}},\\\"status\\\":500}\"}"}

[Mark_Harwood]

The issue is that elasticsearch servers are rejecting requests to prevent overload caused by excessive compilation of user scripts.
I'm not sure where the scripts come from but that is what elasticsearch is objecting to.

[iukea (jordan)]

Is there a way to increase the scripts limit?
Would that help?

[Mark_Harwood]

Yes, the relevant setting is mentioned in the error message.
The error message also warns about the cost of excessive compilations and how to avoid that by using parameters with your scripts

[iukea (jordan)]

Eh the documentation isn't very helpful.

Getting this error from one of Filebeats modules (the Palo Alto one)