Logstash Pipeline crash

Elastic Stack Logstash
1

We are getting some infrequent crashes with logstash. They seem to always be NoMethodFound errors on test_for_inclusion. When it failed no of my error logging is happen so I can not find the location in my pipeline that is causing this

[2024-09-27T02:21:12,123][ERROR][logstash.javapipeline ][corelight_ecs] Pipeline worker error, the pipeline will be stopped {:pipeline_id=>"corelight_ecs", :error=>"(NoMethodError) undefined method `test_for_inclusion' for nil:NilClass", :exception=>Java::OrgJrubyExceptions::NoMethodError, :backtrace=>["usr.share.logstash.vendor.bundle.jruby.$3_dot_1_dot_0.gems.logstash_minus_filter_minus_translate_minus_3_dot_4_dot_2.lib.logstash.filters.translate.filter(/usr/share/logstash/vendor/bundle/jruby/3.1.0/gems/logstash-filter-translate-3.4.2/lib/logstash/filters/translate.rb:261)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.do_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:158)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:176)", "org.jruby.RubyArray.each(org/jruby/RubyArray.java:1989)", "usr.share.logstash.logstash_minus_core.lib.logstash.filters.base.multi_filter(/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:173)", "org.logstash.config.ir.compiler.AbstractFilterDelegatorExt.multi_filter(org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:133)", "RUBY.start_workers(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:304)"], :thread=>"#<Thread:0x245a6985 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 sleep>"}

[2024-09-27T02:21:12,147][WARN ][logstash.javapipeline ][corelight_ecs] Waiting for input plugin to close {:pipeline_id=>"corelight_ecs", :thread=>"#<Thread:0x245a6985 /usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:134 run>"}

[2024-09-27T02:21:14,012][INFO ][logstash.javapipeline ][corelight_ecs] Pipeline terminated {"pipeline.id"=>"corelight_ecs"}

2

Hello,

You need to share your pipeline.

3

Sorry about that here it is

4

My working theory is that we are calling translate for a dictionary work in a few pipelines, and that is the only place I can see that the test_for_inclusion should be called.

5

No doubt. The stacktrace shows the exception is happening at logstash-filter-translate-3.4.2/lib/logstash/filters/translate.rb:261, which is here. @updater has instance scope, so each translate filter has a different value for it. It is initialized in the filter's register method (called once when the pipeline starts). It should never be nil. If one of those calls to SingleValueUpdate.new failed then I would expect the register call to throw an exception and the pipeline should not start.

I cannot think of a way for that exception to happen.

6

To me, it looks like a Logstash bug, not a pipeline bug.