Logstash update log4j version

[mangeshs (Mangesh Shinde)]

I have to change log4j 2.11 to 2.16 so I have replaced all jars.
now its saying

Problems loading a plugin with {:type=>"input", :name=>"beats", :path=>"logstash/inputs/beats", :error_message=>"\n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- org/apache/logging/log4j/log4j-api/2.11.1/log4j-api-2.11.1 (LoadError)", :error_class=>RuntimeError, :error_backtrace=>["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:356:in `do_require'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:265:in `block in require_jar'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:307:in `require_jar_with_block'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/jar_dependencies.rb:264:in `require_jar'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/lib/bootstrap/patches/jar_dependencies.rb:6:in `require_jar'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.1-java/lib/logstash-input-beats_jars.rb:11:in `<main>'", "org/jruby/RubyKernel.java:987:in `require'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/vendor/bundle/jruby/2.5.0/gems/logstash-input-beats-6.0.1-java/lib/logstash/inputs/beats.rb:1:in `<main>'", "org/jruby/RubyKernel.java:987:in `require'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65:in `require'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/plugins/registry.rb:191:in `legacy_lookup'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/plugins/registry.rb:166:in `block in lookup'", "org/jruby/ext/thread/Mutex.java:160:in `synchronize'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/plugins/registry.rb:162:in `lookup'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/plugins/registry.rb:216:in `lookup_pipeline_plugin'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/plugin.rb:143:in `lookup'", "org/logstash/plugins/PluginFactoryExt.java:203:in `plugin'", "org/logstash/plugins/PluginFactoryExt.java:120:in `buildInput'", "org/logstash/execution/JavaBasePipelineExt.java:60:in `initialize'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/pipeline_action/create.rb:36:in `execute'", "C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/agent.rb:326:in `block in converge_state'"]}
[2021-12-15T15:47:26,200][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any input plugin named 'beats'. Are you sure this is correct? Trying to load the beats input plugin resulted in this error: Problems loading the requested plugin named beats of type input. Error: RuntimeError \n\n\tyou might need to reinstall the gem which depends on the missing jar or in case there is Jars.lock then resolve the jars with `lock_jars` command\n\nno such file to load -- org/apache/logging/log4j/log4j-api/2.11.1/log4j-api-2.11.1 (LoadError)", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:100)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:60)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1156)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1143)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.ELK_OSS.logstash_minus_oss_minus_7_dot_4_dot_0.logstash_minus_7_dot_4_dot_0.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/java_pipeline.rb:26)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:332)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:86)", "org.jruby.RubyClass.newInstance(RubyClass.java:915)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "C_3a_.ELK_OSS.logstash_minus_oss_minus_7_dot_4_dot_0.logstash_minus_7_dot_4_dot_0.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/pipeline_action/create.rb:36)", "C_3a_.ELK_OSS.logstash_minus_oss_minus_7_dot_4_dot_0.logstash_minus_7_dot_4_dot_0.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/pipeline_action/create.rb)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:91)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:90)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:183)", "C_3a_.ELK_OSS.logstash_minus_oss_minus_7_dot_4_dot_0.logstash_minus_7_dot_4_dot_0.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/ELK_OSS/logstash-oss-7.4.0/logstash-7.4.0/logstash-core/lib/logstash/agent.rb:326)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:77)", "org.jruby.runtime.Block.call(Block.java:129)", "org.jruby.RubyProc.call(RubyProc.java:295)", "org.jruby.RubyProc.call(RubyProc.java:274)", "org.jruby.RubyProc.call(RubyProc.java:270)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.lang.Thread.run(Unknown Source)"]}

Let me know how to updat gems and resolve this issue ASAP

[yaauie (Ry Biesemeyer)]

The packaged code is still attempting to load the old jars, and is not finding them.

While upgrading the log4j jars and replacing all references in-place may work from a technical standpoint, the recommended mitigations per ESA-2021-31 remain:

@Levine

Users should upgrade to Logstash 7.16.2 or 6.8.22, which were released on December 19, 2021. These releases replace vulnerable versions of Log4j with Log4j 2.17.0.

The widespread flag -Dlog4j2.formatMsgNoLookups=true is NOT sufficient to mitigate the vulnerability in Logstash in all cases, as Logstash uses Log4j in a way where the flag has no effect. If the user cannot upgrade to Logstash 7.16.2 or 6.8.22, it is necessary to remove the JndiLookup class from the log4j2 core jar, with the following command (which is applicable for 5.x, 6.x, and 7.x):

[EDIT: updated with guidance from 2021-12-19 reflecting releases of 7.16.2 and 6.8.22]