Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"no implicit conversion to rational from nil"

Logstash Version : 6.1.1

Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"no implicit conversion to rational from nil"

I am getting this error in Logstash. I am not sure for which logs it is getting error. We are processing millions of records every hour.

I checked my pipeline and tried all possible way to find the problem. But still I can't find.

I also enable log level to TRACE from INFO but that is also not provided any TRACE log.

How I should resolve this issue? It is coming twice/thrice a day and we lose the data because logstash restarted after this error

Hello,

Please share your pipeline so people can see what logstash is trying to do and what could cause the error. Use the preformatted text </> to share it.

It's more than 3k line of filters and that is related to my application which i can't share as public.

If any one need what I am doing in that I can tell.

There should be a stacktrace to show you where in the code that happened. Please post that.

Stacktrace only provided this 2 exceptions

Blockquote[2021-05-31T15:04:18,928][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"no implicit conversion to rational from nil", "backtrace"=>["org/jruby/RubyTime.java:697:in -'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/timestamp.rb:21:in -'", "D:/elk/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-filter-elapsed-4.1.0/lib/logstash/filters/elapsed.rb:161:in filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:164:in block in multi_filter'", "org/jruby/RubyArray.java:1734:in each'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filter_delegator.rb:48:in multi_filter'", "(eval):304590:in block in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):304574:in block in initialize'", "(eval):304716:in block in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):304708:in block in initialize'", "(eval):38028:in block in filter_func'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:455:in filter_batch'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:434:in worker_loop'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:393:in block in start_workers'"], :thread=>"#<Thread:0x268d4f51 sleep>"}

[2021-05-31T15:04:19,100][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<TypeError: no implicit conversion to rational from nil>, :backtrace=>["org/jruby/RubyTime.java:697:in -'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/timestamp.rb:21:in -'", "D:/elk/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-filter-elapsed-4.1.0/lib/logstash/filters/elapsed.rb:161:in filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:164:in block in multi_filter'", "org/jruby/RubyArray.java:1734:in each'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filter_delegator.rb:48:in multi_filter'", "(eval):304590:in block in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):304574:in block in initialize'", "(eval):304716:in block in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):304708:in block in initialize'", "(eval):38028:in block in filter_func'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:455:in filter_batch'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:434:in worker_loop'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:393:in block in start_workers'"]}

Other than this there are debug logs. Like

  1. [DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Grok: adding value to field...
  2. [DEBUG][logstash.pipeline ] filter received...
  3. [DEBUG][logstash.pipeline ] output received...
  4. [DEBUG][logstash.filters.grok ] Event now: {:event=>#LogStash::Event:0x4cfed8ec}
  5. [DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Grok: adding tag
  6. [DEBUG][logstash.filters.grok ] Event now: {:event=>#LogStash::Event:0x1f02f20c}

and many more. According to the other logs which I have seen it has no clue to get where was the error. It does not print any error line number of pipeline. Or Log which has problem.

The exception is occurring in an elapsed filter and one of the events does not have an @timestamp field, probably the start event.