Pipeline aborted due to "Grok::PatternError"

[bo9354]

Configuration OK

Blockquote
Adding pattern {"SYSLOG5424LINE"=>"%{SYSLOG5424BASE} +%{GREEDYDATA:syslog5424_msg}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"}
Adding pattern {"MCOLLECTIVE"=>"., \[%{TIMESTAMP_ISO8601:timestamp} #%{POSINT:pid}\]%{SPACE}%{LOGLEVEL:event_level}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"}
Adding pattern {"MCOLLECTIVEAUDIT"=>"%{TIMESTAMP_ISO8601:timestamp}:", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"}
Pipeline aborted due to error {:exception=>"Grok::PatternError", :backtrace=>["/opt/app/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:123:in `compile'", "org/jruby/RubyKernel.java:1479:in `loop'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:93:in `compile'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in `register'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in `start_workers'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/app/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error, :file=>"logstash/agent.rb", :line=>"493", :method=>"start_pipeline"}
stopping pipeline {:id=>"main", :file=>"logstash/agent.rb", :line=>"406", :method=>"shutdown_pipelines"}
Closing inputs {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"384", :method=>"shutdown"}
stopping {:plugin=>"LogStash::Inputs::File", :level=>:debug, :file=>"logstash/inputs/base.rb", :line=>"81", :method=>"do_stop"}
Closed inputs {:level=>:info, :file=>"logstash/pipeline.rb", :line=>"386", :method=>"shutdown"}

Blockquote

logstash.logstash.conf Configuration.

@bopa

$ cat logstash.logstash.conf

input {
file {
type => "audit"
path => [ "/opt/app/mysql/log/audit.log" ]
start_position => "end"

            codec => multiline {
                    pattern => '^Caused |^Specify |^Dumping |^Unable |^No X11 |^[a-z]|^\t|^\s|^\*|^"|^;|^-<|<\/|^\/|^\(|^<|^< |^> |^>|^ |(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)'
                    what => "previous"
            }

            add_field => {
                    "application" => "Data_360_-_Find"
                    "component" => "MARIA_DB"
                    "environment" => "qa"
            }

    }

}

filter {

    if( [type] == "audit" ){

            grok {
                    match => [ "message", "^%{TIMESTAMP_YMD_TIME:timestamp}%{SPACE}(\[?%{LOGLEVEL:msglevel}\]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})" ]
                    add_field => [ "tags", "grok19701" ]
            } 

            if "_grokparsefailure" in [tags] {
                    grok { 
                            match => [ "message", "^%{TIMESTAMP_DMY_TIME:timestamp}%{SPACE}(\[?%{LOGLEVEL:msglevel}\]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})" ]
                            remove_tag => ["_grokparsefailure"]
                            add_field => [ "msg_level", "INFO" ]
                            add_field => [ "tags", "grok19702" ]
                    }
            }

    }

}

output {
stdout { codec => rubydebug }
}

This seems to be a pattern issue. I cannot put my finger on it.
Please see if you can shed some light on the subject.

[bo9354]

Found this:

Blockquote
Match data {:match=>{"message"=>"^%{TIMESTAMP_YMD_TIME:timestamp}%{SPACE}(\[?%{LOGLEVEL:msglevel}\]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})"}, :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"253", :method=>"register"}
Grok compile {:field=>"message", :patterns=>["^%{TIMESTAMP_YMD_TIME:timestamp}%{SPACE}(\[?%{LOGLEVEL:msglevel}\]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})"], :level=>:info, :file=>"logstash/filters/grok.rb", :line=>"258", :method=>"register"}
regexp: /message {:pattern=>"^%{TIMESTAMP_YMD_TIME:timestamp}%{SPACE}(\[?%{LOGLEVEL:msglevel}\]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})", :level=>:debug, :file=>"logstash/filters/grok.rb", :line=>"260", :method=>"register"}
Adding pattern {"RUUID"=>"\h{32}", :level=>:info, :file=>"grok-pure.rb", :line=>"62", :method=>"add_pattern"}
Blockquote

Pattern file contains:

Blockquote
TIMESTAMP_DMY_TIME %{MONTHDAY}[/| |-]?%{MONTH}[/| |-]?%{YEAR}%{SPACE}%{TIME}
TIMESTAMP_YMD_TIME %{YEAR}[/| |-]?%{MONTHNUM}[/| |-]?%{MONTHDAY}%{SPACE}%{TIME}
Blockquote

All of this works in Grok Debugger:

Blockquote
20180309 12:19:51,zlp25516,solr_ro,zlp25517.vci.att.com,24622,32606111,QUERY,data360,'SELECT * FROM table_core WHERE mots_id='13537' AND schema_name='MSDBO' AND entity_name='MS_SR_ACT_GROUP_AVPN'',0

Blockquote
^%{TIMESTAMP_YMD_TIME:timestamp}%{SPACE}([?%{LOGLEVEL:msglevel}]?)?,?(%{HOSTNAME:hostname},%{WORD:user},%{HOSTNAME:target},%{NUMBER},%{NUMBER},%{WORD:action},%{WORD:database},')%{GREEDYDATA:narrative}?(',%{NUMBER})

Blockquote

[bo9354]

As usual, another dumb mistake.
I left off the patterns_dir:

                    patterns_dir => ["/opt/app/logstash/local/bin/patterns"]

Thanks for your concern. Maybe this will help someone else.