Logstash HTTP API port throwing errors when metricbeat is run

Elastic Stack Logstash
1

Hi! I hope this is the right forum to ask this question.

I have a metricbeat -> logstash -> elasticsearch pipeline. My logstash setup consists of multiple pipelines.

When I run metricbeat (when logstash/elasticsearch are active), I get the following error intermittently:
In metricbeat logs:

{"log.level":"error","@timestamp":"2024-07-02T06:53:56.454-0400","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset logstash.node: HTTP error 500 in : 500 Internal Server Error","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2024-07-02T06:53:56.454-0400","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset logstash.node_stats: HTTP error 500 in : 500 Internal Server Error","service.name":"metricbeat","ecs.version":"1.6.0"}

In logstash logs:

[2024-07-02T06:54:42,294][ERROR][logstash.agent           ] Rack app {:error=>#<NameError: uninitialized constant Rack::Builder::URLMap>, :req=>#<Puma::Client:0x12b8 @ready=true>}
[2024-07-02T06:56:46,980][ERROR][logstash.agent           ] Rack app {:error=>#<NameError: uninitialized constant Rack::Builder::URLMap>, :req=>#<Puma::Client:0x12bc @ready=true>}
[2024-07-02T06:56:46,987][ERROR][logstash.agent           ] Rack app {:error=>#<NameError: uninitialized constant Rack::Builder::URLMap>, :req=>#<Puma::Client:0x12c0 @ready=true>}
[2024-07-02T07:00:20,952][ERROR][logstash.agent           ] Rack app {:error=>#<NameError: uninitialized constant Rack::Builder::URLMap>, :req=>#<Puma::Client:0x12c4 @ready=true>}
[2024-07-02T07:46:24,826][ERROR][logstash.agent           ] Rack app {:error=>#<NameError: uninitialized constant Rack::Builder::URLMap>, :req=>#<Puma::Client:0x12c8 @ready=true>}

What could be the cause of this error? Restarting logstash/metricbeat again gets rid of this error (but not always), but I would preferably like to get rid of it permanently.

The following are my config files:
metricbeat-8.12.1/modules.d/logstash-xpack.yml

- module: logstash
  metricsets: ["node", "node_stats"]
  xpack.enabled: true
  period: 10s
  hosts: ["${PROTOCOL}://${HOSTNAME}:${LS_HTTP_LOW_PORT}"]

logstash.yml

api.http.host: ${HOSTNAME}
api.http.port: ${LS_HTTP_LOW_PORT}-${LS_HTTP_HIGH_PORT}

I have checked, logstash is binding to LS_HTTP_LOW_PORT only and I'm still getting this error.

2

Seems like I'm the error I'm facing is something logstash specific only. I'm getting this error in logstash intermittently, and whenever I do metricbeat doesn't work.

[2024-07-02T11:14:35,374][ERROR][logstash.agent           ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:dlq, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (TypeError) no implicit conversion of nil into String", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1319)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:446)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:92)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:476)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:293)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:840)"]}

I'm getting this error randomly for each of the different pipelines in our config. Any clues to why this could be happening?