I have to parse the java Error logs and pass the result to elastic-search. The pattern that I created is working fine when tested on the site 'https://grokdebug.herokuapp.com/'. However the same grok-pattern when I am copying to the xx-xx.conf file its giving error. I am newbie to ELK stack hence unable to detect issue. Could anyone help.
Below is the content of my file:
input {
file {
path => "C:/Users/xx/Downloads/logstash-tutorial/logstash-tutorial-dataset.log"
start_position => "beginning"
ignore_older => 0
sincedb_path => "nil"
type => "javaStackTrace"
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601}"
negate => true
what => previous}}}
filter {
grok{
match=>{
"message"=>"%{TIMESTAMP_ISO8601:timestamp} %{SPACE}[%{LOGLEVEL:loglevel}]%{SPACE}[%{DATA:thread}] %{SPACE}%{JAVACLASS:class} - %{GREEDYDATA:logmessage}\n%{GREEDYDATA:Exception}%{SPACE}(?m)%{JAVASTACKTRACEPART}"}}}
The error that I am getting when executing the logstash is
[2019-03-07T16:25:51,085][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-03-07T16:25:51,239][ERROR][logstash.pipeline ] Error registering plugin {:pipeline_id=>"main", :plugin=>"#<LogStash::FilterDelegator:0x5471213c @metric_events_out=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: out value:0, @metric_events_in=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: in value:0, @metric_events_time=org.jruby.proxy.org.logstash.instrument.metrics.counter.LongCounter$Proxy2 - name: duration_in_millis value:0, @id="ad9d8390560c5ea3f43e059a39c73a014ed8e17e5df5a8c6c1320088aae6ceab", @klass=LogStash::Filters::Grok, @metric_events=#LogStash::Instrument::NamespacedMetric:0x4edab5d7, @filter=<LogStash::Filters::Grok match=>{"message"=>"%{TIMESTAMP_ISO8601:timestamp} %{SPACE}\\[%{LOGLEVEL:loglevel}\\]%{SPACE}\\[%{DATA:thread}\\] %{SPACE}%{JAVACLASS:class} \\- %{GREEDYDATA:logmessage}\\n%{GREEDYDATA:Exception}%{SPACE}(?m)%{JAVASTACKTRACEPART}"}, id=>"ad9d8390560c5ea3f43e059a39c73a014ed8e17e5df5a8c6c1320088aae6ceab", enable_metric=>true, periodic_flush=>false, patterns_files_glob=>"", break_on_match=>true, named_captures_only=>true, keep_empty_captures=>false, tag_on_failure=>["_grokparsefailure"], timeout_millis=>30000, tag_on_timeout=>"_groktimeout">>", :error=>"target of repeat operator is not specified: /(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))T :?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|+-(?::?(?:(?:[0-5][0-9])))))?) (?:\s)\(?LOGLEVEL:loglevel([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))\\[(?<DATA:thread>.?)\] (?:\s)(?JAVACLASS:class(?:+\.)+[A-Za-z0-9$]+) \- (?GREEDYDATA:logmessage.)\n(?GREEDYDATA:Exception.)(?:\s*)(?m)(?:(?:\s*)at (?JAVACLASS:class(?:+\.)+[A-Za-z0-9$]+)\.(?JAVAMETHOD:method(?:(<(?:cl)?init>)|[a-zA-Z$][a-zA-Z$0-9]*))\((?JAVAFILE:file(?:[A-Za-z0-9.-]+))(?::(?NUMBER:line(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))))))?\))/m", :thread=>"#<Thread:0xbd47599 run>"}
[2019-03-07T16:25:51,244][ERROR][logstash.pipeline ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<RegexpError: target of repeat operator is not specified: /(?<TIMESTAMP_ISO8601:timestamp>(?:(?>\d\d){1,2})-(?:(?:0?[1-9]|1[0-2]))-(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))T :?(?:(?:[0-5][0-9]))(?::?(?:(?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?)))?(?:(?:Z|+-(?::?(?:(?:[0-5][0-9])))))?) (?:\s*)[(?LOGLEVEL:loglevel([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))](?:\s*)[(?<DATA:thread>.?)] (?:\s)(?JAVACLASS:class(?:+.)+[A-Za-z0-9$]+) - (?GREEDYDATA:logmessage.)\n(?GREEDYDATA:Exception.)(?:\s*)(?m)(?:(?:\s*)at (?JAVACLASS:class(?:+.)+[A-Za-z0-9$]+).(?JAVAMETHOD:method(?:(<(?:cl)?init>)|[a-zA-Z$][a-zA-Z$0-9]*))((?JAVAFILE:file(?:[A-Za-z0-9.-]+))(?::(?NUMBER:line(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:.[0-9]+)?)|(?:.[0-9]+)))))))?))/m>, :backtrace=>["org/jruby/RubyRegexp.java:928:in initialize'", "C:/Users/Documents/logstash-6.5.1/vendor/bundle/jruby/2.3.0/gems/jls-grok-0.11.5/lib/grok-pure.rb:127:incompile'", "C:/Users/Documents/logstash-6.5.1/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:281:in block in register'", "org/jruby/RubyArray.java:1734:ineach'", "C:/Users/Documents/logstash-6.5.1/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:275:in block in register'", "org/jruby/RubyHash.java:1343:ineach'", "C:/Users/Documents/logstash-6.5.1/vendor/bundle/jruby/2.3.0/gems/logstash-filter-grok-4.0.4/lib/logstash/filters/grok.rb:270:in register'", "C:/Users/Documents/logstash-6.5.1/logstash-core/lib/logstash/pipeline.rb:242:inregister_plugin'", "C:/Users/Documents/logstash-6.5.1/logstash-core/lib/logstash/pipeline.rb:253:in block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "C:/Users/Documents/logstash-6.5.1/logstash-core/lib/logstash/pipeline.rb:253:in register_plugins'", "C:/Users/Documents/logstash-6.5.1/logstash-core/lib/logstash/pipeline.rb:595:inmaybe_setup_out_plugins'", "C:/Users/Documents/logstash-6.5.1/logstash-core/lib/logstash/pipeline.rb:263:in start_workers'", "C:/Users/Documents/logstash-6.5.1/logstash-core/lib/logstash/pipeline.rb:200:inrun'", "C:/Users/Documents/logstash-6.5.1/logstash-core/lib/logstash/pipeline.rb:160:in `block in start'"], :thread=>"#<Thread:0xbd47599 run>"}
[2019-03-07T16:25:51,269][ERROR][logstash.agent ] Failed to execute action {:id=>:main, :action_type=>LogStash::ConvergeResult::FailedAction, :message=>"Could not execute action: PipelineAction::Create, action_result: false", :backtrace=>nil}
[2019-03-07T16:25:51,706][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
Sample Log I tested in the above mentioned url:
2019-01-09 15:00:05.800 [ERROR] [http-nio-9096-exec-1] com.exapmle.exception.GlobalExceptionFacade - Application encountered as unexpected error
java.lang.NullPointerException: null
at com.exapmle.service.impl.DashboardOverviewServiceImpl.contBusinessSpend(DashboardOverviewServiceImpl.java:2149)