Logstash not working

Hello,

My logstash instance which is integrated with Azure Sentinel was working well with out any error (All the pipelines were functional, and the events were received at Azure Sentinel). A while ago, I made a couple of configuration changes on a conf file (/etc/logstash/conf.d/), and I restarted in service, since then I am getting the following error, please advise.

Version of Logstash: 7.17 (I can't use the latest version of Logstash since it is not supported by Azure Sentinel)
No other change is performed, except updating a conf file.

[ERROR] 2022-09-09 17:41:49.257 [Converge PipelineAction::Create<main>] registry - Unable to load plugin. {:type=>"output", :name=>"microsoft-logstash-output-azure-loganalytics"}
[ERROR] 2022-09-09 17:41:49.266 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any output plugin named 'microsoft-logstash-output-azure-loganalytics'. Are you sure this is correct? Trying to load the microsoft-logstash-output-azure-loganalytics output plugin resulted in this error: Unable to load the requested plugin named microsoft-logstash-output-azure-loganalytics of type output. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:388)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
          create at org/logstash/execution/ConvergeResultExt.java:135
             add at org/logstash/execution/ConvergeResultExt.java:60
  converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:401
[ERROR] 2022-09-09 17:41:49.270 [Agent thread] agent - An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
[FATAL] 2022-09-09 17:41:49.277 [LogStash::Runner] runner - An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:401:in `block in converge_state'"]}
[FATAL] 2022-09-09 17:41:49.282 [LogStash::Runner] Logstash - Logstash stopped processing because of an error: (SystemExit) exit
org.jruby.exceptions.SystemExit: (SystemExit) exit
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
        at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]

--
Thanks,
Siddarth

What do you get if you do

cd /etc/logstash ; bin/logstash-plugin list | grep '*azure*'

I don't find Azure.

It's strange that the plugin is apparently missing.
And this events has simultaneously occured on 2 servers.

--
Thanks,
Siddarth

@Badger , do you have any solution for this. Please advise.

--
Thanks,
Siddarth

What does your configuration looks like? What is your output?

This plugin microsoft-logstash-output-azure-loganalytics is not bundled per default, it is a third party plugin, here is the github repository.

In the README.md file in that repository it says:

Required Logstash version: between 7.0 and 7.16

Did you have this running before in which version of Logstash? It looks like your logstash was updated and the plugin wasn't reinstalled or do not work with this version.

@leandrojmp , yes it was working until Friday noon. Since Friday noon it stopped working. The system was functional since last 3 months .

We have 2 such servers. Both were working fine, and since Friday noon, both have stopped working.

The current version shows 7.17. I am not sure which version was installed. How do I confirm if an update occurred. Please advise.

--
Thanks,
Siddarth

Are you the only one with access to those servers? Maybe somebody else updated the system.

You will need to check your systems logs.

@leandrojmp , Logtash 7.15 was installed which later got updated to 7.17. is there a way to downgrade the version?
When I try to follow the documentation, the version available is 7.17 which is not supported.

--
Thanks,
Siddarth

You will need to remove Logstash 7.17 and reinstall the specific version.

How you will do this depends in your system.

For Red Hat Based is something like this:

yum remove logstash
yum install logstash-7.15.2

Just check how you remove and reinstall a package in your system.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.