Too many dynamic script compilations within one minute, max: [15/min];

[varun_kumar2 (Varun Kumar)]

Here is the output with error
{
"_id": "CRM.Email.Monitor.Errors_d5a3faac-50c2-4238-929b-4e84088f9fd4-2017-10-25T12:36:16.292Z",
"watch_record": {
"watch_id": "CRM.Email.Monitor.Errors",
"state": "failed",
"_status": {
"state": {
"active": true,
"timestamp": "2017-10-25T12:36:10.942Z"
},
"actions": {
"send_email": {
"ack": {
"timestamp": "2017-10-25T12:36:10.942Z",
"state": "awaits_successful_execabcon"
}
}
}
},
"trigger_event": {
"type": "manual",
"triggered_time": "2017-10-25T12:36:16.292Z",
"manual": {
"schedule": {
"scheduled_time": "2017-10-25T12:36:16.292Z"
}
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logstash-"
],
"types": [
"logevent"
],
"body": {
"query": {
"bool": {
"must": [
{
"match": {
"level": "Error"
}
},
{
"match": {
"fields.ApplicationSuite": "CRM"
}
},
{
"match": {
"fields.ApplicationName": "CRMEmailMonitor"
}
},
{
"match_phrase_prefix": {
"message": "There are Failed CRM Emails"
}
}
],
"filter": {
"range": {
"@timestamp": {
"from": "now-16m",
"to": "now"
}
}
}
}
},
"script_fields": {
"TimeStampFormatted": {
"script": {
"inline": "def sf = new SimpleDateFormat('yyyy-MM-dd HH:mm:ss');String dateStr = sf.format(doc['@timestamp'].value);"
}
},
"LastEmailSentFormatted": {
"script": {
"inline": "def sf = new SimpleDateFormat('yyyy-MM-dd HH:mm:ss');String dateStr = sf.format(doc['fields.LastEmailSent'].value);"
}
},
"ApplicationName": {
"script": "_source.fields.ApplicationName"
},
"FailedEmailCount": {
"script": "_source.fields.FailedEmailCount"
}
},
"sort": [
{
"@timestamp": "desc"
}
],
"size": 100
}
}
}
},
"condition": {
"script": {
"inline": "def sf = new SimpleDateFormat('yyyy-MM-dd HH:mm:ss');String dateStr = sf.format(new Date());def dt = sf.parse(dateStr);Calendar cal = sf.getCalendar(); cal.setTime(dt);cal.add(Calendar.MINUTE, -16);def instant = cal.toInstant();def localDateTime = LocalDateTime.ofInstant(instant, ZoneOffset.UTC);ctx.vars.from = localDateTime;return ctx.payload.hits.total>0",
"lang": "painless"
}
},
"result": {
"execabcon_time": "2017-10-25T12:36:16.292Z",
"execabcon_duration": 907,
"input": {
"type": "search",
"status": "failure",
"reason": "SearchPhaseExecabconException[all shards failed]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_per_minute] setting]; ",
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logstash-"
],
"types": [
"logevent"
],
"body": {
"query": {
"bool": {
"must": [
{
"match": {
"level": "Error"
}
},
{
"match": {
"fields.ApplicationSuite": "CRM"
}
},
{
"match": {
"fields.ApplicationName": "CRMEmailMonitor"
}
},
{
"match_phrase_prefix": {
"message": "There are Failed CRM Emails"
}
}
],
"filter": {
"range": {
"@timestamp": {
"from": "now-16m",
"to": "now"
}
}
}
}
},
"script_fields": {
"TimeStampFormatted": {
"script": {
"inline": "def sf = new SimpleDateFormat('yyyy-MM-dd HH:mm:ss');String dateStr = sf.format(doc['@timestamp'].value);"
}
},
"LastEmailSentFormatted": {
"script": {
"inline": "def sf = new SimpleDateFormat('yyyy-MM-dd HH:mm:ss');String dateStr = sf.format(doc['fields.LastEmailSent'].value);"
}
},
"ApplicationName": {
"script": "_source.fields.ApplicationName"
},
"FailedEmailCount": {
"script": "_source.fields.FailedEmailCount"
}
},
"sort": [
{
"@timestamp": "desc"
}
],
"size": 100
}
}
}
},
"actions": []
},
"messages": [
"failed to execute watch input"
]
}
}

[spinscale (Alexander Reelsen)]

the exception reason already states what to do: set script.max_compilations_per_minute to a higher value - I suppose you have several watches, that contain several scripts, also note that a few things, that internal are treated as a script are hard to spot, for example the whole request body of a search input.

Hope this helps!

--Alex