Hi All,
I have noticed that Logstash will stop and not recover if a grok pattern is reference from a file that does not exist.
Currently we use an git repo to manage our grok patterns in a set of folders & files. Once changes are committed a trigger runs and a build occurs that ultimately pushes to S3. We then have a systemd process running every 5 minutes consuming files from S3 onto the EC2 instance.
We are then using logstash pipelines in Kibana to update the logstash patterns.
Today, a scenario occurred whereby the built trigger didn't occur yet - but the patterns were updated in logstash central pipeline. We then observed that no logs were being pushed to elastic. Upon looking in logstash-plain.log the following entries was found:
[2019-04-18T17:07:18,167][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"beat-ingestion", :exception=>#<Grok::PatternError: pattern %{PROCESS_ORDER} not defined>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.4/lib/grok-pure.rb:123"}
Even after manually running build triggers, file being on S3 and pushed to EC2 the logstash process never recovered and needed a restart.
Is there a configuration I am missing to ensure that a missing file/pattern doesn't cause logstash to abort?
Below is my current filter configuration:
filter {
if [app_name] == "mspbatchsubscriptions" {
grok {
patterns_dir => ["/opt/grokpatterns/mspbatchsubscriptions"]
match => { "message" => "%{DEFAULT}" }
match => { "message" => "%{MSP_BATCH_RETRY}" }
}
}
if [app_name] == "ruppifs" {
grok {
patterns_dir => ["/opt/grokpatterns/ruppifs/"]
match => { "message" => "%{PROCESS_ORDER}" }
match => { "message" => "%{TIME_TAKEN}" }
match => { "message" => "%{GET_CUSTOMER_MANAGER}" }
match => { "message" => "%{REST_ERROR_RESPONSE}" }
match => { "message" => "%{UNITED_MILEAGE}" }
match => { "message" => "%{SOAP_ERRORS_CODE_TEXT}" }
match => { "message" => "%{SOAP_ERRORS_CODE_MSG}" }
match => { "message" => "%{ACTIVATION_SUCCESS}" }
match => { "message" => "%{AUTHENTICATION_SUCCESS}" }
match => { "message" => "%{SOAP_PROCESS_ORDER}" }
match => { "message" => "%{FORGOT_PASSWORD}" }
match => { "message" => "%{ACCOUNT_RESET}" }
match => { "message" => "%{ACTIVATING_CUSTOMER}" }
match => { "message" => "%{REMEMBER_ME}" }
}
}
}
Thank you as always.
Wayne