Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"no implicit conversion to rational from nil"

Elastic Stack Logstash
1

Logstash Version : 6.1.1

Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"no implicit conversion to rational from nil"

I am getting this error in Logstash. I am not sure for which logs it is getting error. We are processing millions of records every hour.

I checked my pipeline and tried all possible way to find the problem. But still I can't find.

I also enable log level to TRACE from INFO but that is also not provided any TRACE log.

How I should resolve this issue? It is coming twice/thrice a day and we lose the data because logstash restarted after this error

2

Hello,

Please share your pipeline so people can see what logstash is trying to do and what could cause the error. Use the preformatted text </> to share it.

3

It's more than 3k line of filters and that is related to my application which i can't share as public.

If any one need what I am doing in that I can tell.

4

There should be a stacktrace to show you where in the code that happened. Please post that.

5

Stacktrace only provided this 2 exceptions

Blockquote[2021-05-31T15:04:18,928][ERROR][logstash.pipeline ] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline_id=>"main", "exception"=>"no implicit conversion to rational from nil", "backtrace"=>["org/jruby/RubyTime.java:697:in -'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/timestamp.rb:21:in -'", "D:/elk/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-filter-elapsed-4.1.0/lib/logstash/filters/elapsed.rb:161:in filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:164:in block in multi_filter'", "org/jruby/RubyArray.java:1734:in each'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filter_delegator.rb:48:in multi_filter'", "(eval):304590:in block in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):304574:in block in initialize'", "(eval):304716:in block in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):304708:in block in initialize'", "(eval):38028:in block in filter_func'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:455:in filter_batch'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:434:in worker_loop'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:393:in block in start_workers'"], :thread=>"#<Thread:0x268d4f51 sleep>"}

[2021-05-31T15:04:19,100][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<TypeError: no implicit conversion to rational from nil>, :backtrace=>["org/jruby/RubyTime.java:697:in -'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/timestamp.rb:21:in -'", "D:/elk/logstash-6.1.1/vendor/bundle/jruby/2.3.0/gems/logstash-filter-elapsed-4.1.0/lib/logstash/filters/elapsed.rb:161:in filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:145:in do_filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:164:in block in multi_filter'", "org/jruby/RubyArray.java:1734:in each'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filters/base.rb:161:in multi_filter'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/filter_delegator.rb:48:in multi_filter'", "(eval):304590:in block in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):304574:in block in initialize'", "(eval):304716:in block in initialize'", "org/jruby/RubyArray.java:1734:in each'", "(eval):304708:in block in initialize'", "(eval):38028:in block in filter_func'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:455:in filter_batch'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:434:in worker_loop'", "D:/elk/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:393:in block in start_workers'"]}

Other than this there are debug logs. Like

  1. [DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Grok: adding value to field...
  2. [DEBUG][logstash.pipeline ] filter received...
  3. [DEBUG][logstash.pipeline ] output received...
  4. [DEBUG][logstash.filters.grok ] Event now: {:event=>#LogStash::Event:0x4cfed8ec}
  5. [DEBUG][logstash.util.decorators ] filters/LogStash::Filters::Grok: adding tag
  6. [DEBUG][logstash.filters.grok ] Event now: {:event=>#LogStash::Event:0x1f02f20c}

and many more. According to the other logs which I have seen it has no clue to get where was the error. It does not print any error line number of pipeline. Or Log which has problem.

6

The exception is occurring in an elapsed filter and one of the events does not have an @timestamp field, probably the start event.

7

I resolved this issue. I need to reduce the load on the logstash and it started working

Earlier I was running 10 pipeline workers with 5k logs per worker. After reducing this to 1 pipeline worker and 2k logs per worker. It started working.

I was having good config server of AWS c5.4xlarge instance and provided good amount of ram to logstash by updating jvm config of logstash.

Logstash should provide proper info. I was looking at my logs and was blocked finding that why it is not working. Later I reverted my last change of increasing the pipeline worker and logs per worker and it started working.

Thanks everyone for you answer!!!

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.