Logstash to panos pipeline - CircuitBreakerException script.max_compilations_rate

I am using the filebeat panw module to ingest palo-alto logs. Due to some performance testing the current path is a little convoluted, but filebeat listens on syslog port with panw module and the output sends to Redis. Then logstash servers have Redis inputs to send to elasticsearch.
That final logstash pipeline config looks like this:

input 
{
	redis 
	{
		host => "x.x.x.x"
		port => xxxx
		data_type => "list"
		key => "xxxx"
		password => "xxxx"
		codec => json
		id => "input_redis_pa_test2"
	}
}
filter
{
}
output 
{
	elasticsearch 
	{
		hosts => ["https://xxxx.com:9200"]
		index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}-000001"
		user => "xxxx"
		password => "xxxx"
		ssl => true
		cacert => 'xxxx'
		id => "output_elasticsearch_pa_test4"
		pipeline => "filebeat-7.7.0-panw-panos-pipeline"
	}
}

Logstash is reporting the following error:

[2020-08-24T15:30:51,190][INFO ][logstash.outputs.elasticsearch][palo-alto_ingest_redis_test4_pipeline][output_elasticsearch_pa_test4] retrying failed action with response code: 500 ({"type"=>"illegal_state_exception", "reason"=>"pipeline with id [filebeat-7.7.0-panw-panos-pipeline] could not be loaded, caused by [ElasticsearchParseException[Error updating pipeline with id [filebeat-7.7.0-panw-panos-pipeline]]; nested: GeneralScriptException[Failed to compile inline script [long nanos = ctx['event']['duration'] * params.NANOS_IN_A_SECOND; ctx['event']['duration'] = nanos; def start = ctx.event?.start; if (start != null) {\n ctx.event['end'] = ZonedDateTime.parse(start).plusNanos(nanos);\n}\n] using lang [painless]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; GeneralScriptException[Failed to compile inline script [long nanos = ctx['event']['duration'] * params.NANOS_IN_A_SECOND; ctx['event']['duration'] = nanos; def start = ctx.event?.start; if (start != null) {\n ctx.event['end'] = ZonedDateTime.parse(start).plusNanos(nanos);\n}\n] using lang [painless]]; nested: CircuitBreakingException[[script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting];; org.elasticsearch.common.breaker.CircuitBreakingException: [script] Too many dynamic script compilations within, max: [75/5m]; please use indexed, or scripts with parameters instead; this limit can be changed by the [script.max_compilations_rate] setting]"})

This error is being repeated many times per second.
The panw logs appear to be all ingesting, but maybe this represents some logs that are not making it there. I'm not sure.
I'm surprised to get this error given the "filebeat-7.7.0-panw-panos-pipeline" is a built-in pipeline, not a custom one. From what I've read, in creasing the limit is not necessarily the best way to handle this, but I tried a 20/1m and a few other settings:

PUT /_cluster/settings
{
    "transient" : {
        "script.max_compilations_rate" : "20/1m"
    }
}

Whether unrelated or not, when I do this, for a few seconds afterwards the error being spammed changes to this:

[2020-08-24T15:52:03,776][WARN ][logstash.outputs.elasticsearch][palo-alto_ingest_redis_test4_pipeline][output_elasticsearch_pa_test4] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"filebeat-7.7.0-2020.08.24-000001", :routing=>nil, :_type=>"_doc", :pipeline=>"filebeat-7.7.0-panw-panos-pipeline"}, #<LogStash::Event:0x322dfeb7>], :response=>{"index"=>{"_index"=>"filebeat-7.7.0-2020.08.24-000001", "_type"=>"_doc", "_id"=>"-w0HInQBlL-ZnZn2EB7z", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [event.created] of type [date] in document with id '-w0HInQBlL-ZnZn2EB7z'. Preview of field's value: '2020/08/24 19:51:53'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"failed to parse date field [2020/08/24 19:51:53] with format [strict_date_optional_time||epoch_millis]", "caused_by"=>{"type"=>"date_time_parse_exception", "reason"=>"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}}}}

Then after a few seconds it starts repeating the original error message containing 75/5m. Which is also surprising since I've supposedly now set it to 20/1m.

If there's something I can do to help pinpoint what is causing the error that would be helpful.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.