Hi,
I am trying to parse really large multiline logs, the file is first ingested using filebeat. There i have specified multiline settings so it seems to be fine over ther. However, while ingesting files in logstash an error is thrown:
[2019-05-22T16:41:24,490][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ThreadError: interrupted in Thread::Queue#pop>, :backtrace=>["org/jruby/ext/thread/Queue.java:344:in pop'", "/Users/vivekvardhan.reddy/Applications/logstash/vendor/bundle/jruby/2.5.0/gems/stud-0.0.23/lib/stud/task.rb:38:in
wait'", "/Users/vivekvardhan.reddy/Applications/logstash/logstash-core/lib/logstash/runner.rb:379:in execute'", "/Users/vivekvardhan.reddy/Applications/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:67:in
run'", "/Users/vivekvardhan.reddy/Applications/logstash/logstash-core/lib/logstash/runner.rb:237:in run'", "/Users/vivekvardhan.reddy/Applications/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-0.6.5/lib/clamp/command.rb:132:in
run'", "/Users/vivekvardhan.reddy/Applications/logstash/lib/
Here is the part of my config which seems to be the source of issue as when i commented out this code the pipeline was running fine and not throwing this error.
if [message] =~ ".*id: nbnplus.*" {
grok {
match =>
{
message => "%{DATA:thread} %{DATA:sessionid}\s(txid\=%{BASE16NUM})?\s?Time:\s+%{TIME:totaltime}\sid\s?:\s?%{PROG:id}(\[%{DATA:glidefunc}\])?\s?for:\s(?<querybody>(.|\r|\n)*)"
}
tag_on_failure => "_nbnplus_grokparsefailure"
}
if [totaltime] {
dissect {
mapping => {
"totaltime" => "%{hour}:%{minute}:%{second}.%{millisecond}"
}
}
if [hour] and [minute] and [second] and [millisecond] {
ruby {
code => "event.set( 'totaltimeMS', (event.get('hour').to_i*3600*1000 + event.get('minute').to_i*60*10000 + event.get('second').to_i*1000 + event.get('millisecond').to_i))"
}
mutate {
remove_field => ["hour","minute","second","millisecond"]
}
}
}
}
Could anyone please point me in the right direction?
Thanks in advance,
Vivek