Hi everyone,
I'm struggling with a strange problem with a Logstash 5 that I'm running on a Windows 2012 x64.... Logstash seems to miss the output of some events, or sometimes it stops outputting events entirely.
My Logstash is getting its input from a PostgreSQL database, it filters/builds the events, and them outputs them via HTTP. The Logstash configuration is like this:
input {
jdbc {
...
statement => "select * from metrics_table where metric_timestamp > :sql_last_value"
use_column_value => "true"
tracking_column => "metric_timestamp"
tracking_column_type => "numeric"
last_run_metadata_path => "../log/metrics-logstash_jdbc_last_run"
schedule => "* * * * *"
....
}
}
filter {
## Add fields, rename fields, remove fields, convert some fields, and then...
ruby {
code => "
event.to_hash.each do |k,v|
if v == nil
event.remove(k)
end
end
event.set( 'event_body', event.to_json.inspect )
event.to_hash.each do |k|
if k !~ /m_Timestamp|^m_CustomerRelay$|event_body/
event.remove(k)
end
end
"
}
}
output {
http {
http_method => "post"
content_type => "application/json"
format => "message"
...
message => '{ "time":"%{m_Timestamp}", "host":"%{m_CustomerRelay}", "event":%{event_body} }'
}
}
The database table is generating around 350 new entries per minute, and the table only has the data for the current day.
Everything starts an seem to be working as expected, but them it seems that after some time Logstatsh stops outputting the events, and I also detected that some events present in the DB table are never sent via teh HTTP output.
When I activate debug in Logstash, the debug output is very heavy, but I've only detected errors similar to the one bellow for now:
[2017-07-17T11:21:14,514][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<ThreadError: killed thread>, :backtrace=>["org/jruby/RubyThread.java:836:inwakeup'", "C:/PROGRA~2/ALTITU~1/LOGSTA~1/ahm-metrics/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/interval.rb:44:in stop!'", "C:/PROGRA~2/ALTITU~1/LOGSTA~1/ahm-metrics/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/task.rb:52:instop!'", "C:/PROGRA~2/ALTITU~1/LOGSTA~1/ahm-metrics/logstash-core/lib/logstash/runner.rb:411:in trap_sigint'", "org/jruby/RubyProc.java:281:incall'", "org/jruby/RubyArray.java:1613:in each'", "C:/PROGRA~2/ALTITU~1/LOGSTA~1/ahm-metrics/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/trap.rb:46:insimulate_signal'", "C:/PROGRA~2/ALTITU~1/LOGSTA~1/ahm-metrics/vendor/bundle/jruby/1.9/gems/stud-0.0.22/lib/stud/trap.rb:26:in trap'", "org/jruby/RubyProc.java:281:incall'"]}`
I have a second Logstash agent with a very similar configuration running on the same Windows machine, receiving its input from a second database table, and this one keeps working and working as expected.
The only thing different is that the event rate in that case is significantly lower, about 50 new lines per minute.
Any pointers for the cause or to aid me in the investigation?
Thanks.