Pipeline aborted due to "Grok::PatternError"


#1

Configuration OK

Blockquote
Adding pattern {"SYSLOG5424LINE"=>"%{SYSLOG5424BASE} +%{GREEDYDATA:syslog5424_msg}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"}
Adding pattern {"MCOLLECTIVE"=>"., \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\]%{SPACE}%{LOGLEVEL:event_level}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"}
Adding pattern {"MCOLLECTIVEAUDIT"=>"%{TIMESTAMP_ISO8601:timestamp}:", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"}
Pipeline aborted due to error {:exception=>"Grok::PatternError", :backtrace=>["/opt/app/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:123:in compile'", "org/jruby/RubyKernel.java:1479:inloop'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:93:in compile'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:inregister'", "org/jruby/RubyArray.java:1613:in each'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:inregister'", "org/jruby/RubyHash.java:1342:in each'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:inregister'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in start_workers'", "org/jruby/RubyArray.java:1613:ineach'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in start_workers'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:inrun'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error, :file=>"logstash/agent.rb", :line=>"493", :method=>"start_pipeline"}
stopping pipeline {:id=>"main", :file=>"logstash/agent.rb", :line=>"406", :method=>"shutdown_pipelines"}
Closing inputs {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"384", :method=>"shutdown"}
stopping {:plugin=>"LogStash::Inputs::File", :level=>:debug, :file=>"logstash/inputs/base.rb", :line=>"81", :method=>"do_stop"}
Closed inputs {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"386", :method=>"shutdown"}

Blockquote

logstash.logstash.conf Configuration.

This seems to be a pattern issue. I cannot put my finger on it.
Please see if you can shed some light on the subject.


#2

Found this:

Blockquote
Match data {:match=>{"message"=>"^%{TIMESTAMP_YMD_TIME:timestamp}%{SPACE}(\[?%{LOGLEVEL:msglevel}\]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})"}, :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"253", :method=>"register"}
Grok compile {:field=>"message", :patterns=>["^%{TIMESTAMP_YMD_TIME:timestamp}%{SPACE}(\[?%{LOGLEVEL:msglevel}\]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})"], :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"258", :method=>"register"}
regexp: /message {:pattern=>"^%{TIMESTAMP_YMD_TIME:timestamp}%{SPACE}(\[?%{LOGLEVEL:msglevel}\]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})", :level=>:debug, :file=>"logstash/filters/grok.rb", :line=>"260", :method=>"register"}
Adding pattern {"RUUID"=>"\h{32}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"}
Blockquote

Pattern file contains:

Blockquote
TIMESTAMP_DMY_TIME %{MONTHDAY}[/| |-]?%{MONTH}[/| |-]?%{YEAR}%{SPACE}%{TIME}
TIMESTAMP_YMD_TIME %{YEAR}[/| |-]?%{MONTHNUM}[/| |-]?%{MONTHDAY}%{SPACE}%{TIME}
Blockquote

All of this works in Grok Debugger:

Blockquote
20180309 12:19:51,zlp25516,solr_ro,zlp25517.vci.att.com,24622,32606111,QUERY,data360,'SELECT * FROM table_core WHERE mots_id='13537' AND schema_name='MSDBO' AND entity_name='MS_SR_ACT_GROUP_AVPN'',0

Blockquote
^%{TIMESTAMP_YMD_TIME:timestamp}%{SPACE}([?%{LOGLEVEL:msglevel}]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})

Blockquote


#3

As usual, another dumb mistake.
I left off the patterns_dir:

                    patterns_dir => ["/opt/app/logstash/local/bin/patterns"]

Thanks for your concern. Maybe this will help someone else.


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.