Logstash Stops if Grok Patterns file/pattern does not exist - Logstash 6.3.2

Hi All,

I have noticed that Logstash will stop and not recover if a grok pattern is reference from a file that does not exist.

Currently we use an git repo to manage our grok patterns in a set of folders & files. Once changes are committed a trigger runs and a build occurs that ultimately pushes to S3. We then have a systemd process running every 5 minutes consuming files from S3 onto the EC2 instance.

We are then using logstash pipelines in Kibana to update the logstash patterns.

Today, a scenario occurred whereby the built trigger didn't occur yet - but the patterns were updated in logstash central pipeline. We then observed that no logs were being pushed to elastic. Upon looking in logstash-plain.log the following entries was found:

[2019-04-18T17:07:18,167][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"beat-ingestion", :exception=>#<Grok::PatternError: pattern %{PROCESS_ORDER} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.4/lib/grok-pure.rb:123"}

Even after manually running build triggers, file being on S3 and pushed to EC2 the logstash process never recovered and needed a restart.

Is there a configuration I am missing to ensure that a missing file/pattern doesn't cause logstash to abort?

Below is my current filter configuration:

filter {
  if [app_name] == "mspbatchsubscriptions" {
    grok {
       patterns_dir => ["/opt/grokpatterns/mspbatchsubscriptions"] 
       match => { "message" => "%{DEFAULT}" }
       match => { "message" => "%{MSP_BATCH_RETRY}" }
    }
  }
  if [app_name] == "ruppifs" {
    grok {
       patterns_dir => ["/opt/grokpatterns/ruppifs/"]
       match => { "message" => "%{PROCESS_ORDER}" }
       match => { "message" => "%{TIME_TAKEN}" }
       match => { "message" => "%{GET_CUSTOMER_MANAGER}" }
       match => { "message" => "%{REST_ERROR_RESPONSE}" }
       match => { "message" => "%{UNITED_MILEAGE}" }
       match => { "message" => "%{SOAP_ERRORS_CODE_TEXT}" }
       match => { "message" => "%{SOAP_ERRORS_CODE_MSG}" }
       match => { "message" => "%{ACTIVATION_SUCCESS}" }
	   match => { "message" => "%{AUTHENTICATION_SUCCESS}" }
       match => { "message" => "%{SOAP_PROCESS_ORDER}" }
       match => { "message" => "%{FORGOT_PASSWORD}" }
       match => { "message" => "%{ACCOUNT_RESET}" }
       match => { "message" => "%{ACTIVATING_CUSTOMER}" }
       match => { "message" => "%{REMEMBER_ME}" }
    }
  }
}

Thank you as always.

Wayne

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.