Logstash crashes at startup when custom codec is used in the config

Elastic Stack Logstash
1

Hi Team

I'm participating in the migration of a project from Logstash 6.4.3 to Logstash 7.10.2. The problem is that the new Logstash crashes at startup when one of our custom codecs is used in the configuration.

The error from the logstash-plain.log (formatted for readability):

Failed to execute action {
  :action=>LogStash::PipelineAction::Create/pipeline_id:main,
  :exception=>"Java::JavaLang::IllegalStateException",
  :message=>"Unable to configure plugins: Missing Converter handling for full class name=org.jruby.gen.u20, simple name=RubyObject20",
  :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)",
  "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:83)",
  "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)",
  "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)",
  "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)",
  "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:84)",
  "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)",
  "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)",
  "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:86)",
  "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:73)",
  "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)",
  "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)",
  "org.jruby.RubyClass.newInstance(RubyClass.java:939)",
  "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)",
  "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)",
  "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)",
  "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:549)",
  "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:361)",
  "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)",
  "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:92)",
  "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:191)",
  "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:178)",
  "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:208)",
  "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:396)",
  "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:205)",
  "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:325)",
  "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:72)",
  "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)",
  "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:137)",
  "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:60)",
  "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)",
  "org.jruby.runtime.Block.call(Block.java:139)",
  "org.jruby.RubyProc.call(RubyProc.java:318)",
  "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)",
  "java.lang.Thread.run(Thread.java:748)"]
}

An exception happened when converging configuration {
  :exception=>LogStash::Error,
  :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`",
  :backtrace=>["org/logstash/execution/ConvergeResultExt.java:129:in `create'",
  "org/logstash/execution/ConvergeResultExt.java:57:in `add'",
  "/opt/logstash/logstash-core/lib/logstash/agent.rb:378:in `block in converge_state'"]
}

Here is a minimal Logstash config with which I was able to reproduce the issue:

input {
  file {
    path   => [ "/var/log/logstash/chain-input.txt" ]
    codec  => "json"
  }
}

output {
  file {
    path => "/var/log/logstash/chain-output.log"
    codec => chain {
      codec0 => json {}
    }
  }
}

As you can see the config uses "chain" codec which is developed internally in the company where I work. It was implemented 5 years ago and was working with previous Logstash versions (2..6) without problems.

How can I dig deeper into the issue? How to find out the line in the codec which causes the issue?

Or maybe it is a problem with Logstash or JRuby? How can I confirm it?

2

I've discovered that if I run Logstash with the parameter --java-execution false then it starts successfully with the same configuration. Which leads me to a conclusion that it is a Logstash issue. Any thoughts or suggestions?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.