Multiple pipelines: StringIndexOutOfBoundsException: String index out of range: -1

Elastic Stack Logstash
1

Hello everybody,

i recently set up an ELK stack and am running Logstash on a Windows host which processes input from log files and sends them to elasticsearch on another host.

I use the "collector pattern" to process two different log files with different filters and send them to the "collector" pipeline which outputs the results to elasticsearch.

So there are 3 pipelines.
When i use only the output pipeline and another one of those 2 file processing pipelines, all is fine.

When i enable both, i SOMETIMES (as in every 5th try or so), when starting Logstash, i get an error:

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:wms-wmspipeline, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (LoadError) load error: cabin/outputs/stdlib-logger -- java.lang.StringIndexOutOfBoundsException: String index out of range: -1"

The error always happens at the wms-wms pipeline!

It is not always cabin/outputs/stdlib-logger, but whenever i comment out some stuff in the pipeline configuration, it is the same StringIndexOutOfBoundsException, but related to grok-pure or cabin/mixins/timer.

Please see the full config and strack trace here:


pipelines.yml
=============
 
- pipeline.id: wms-pdainteractionpipeline
path.config: "config/wms-pdainteraction.conf"
 
- pipeline.id: wms-wmspipeline
path.config: "config/wms-wms.conf"
 
- pipeline.id: es
path.config: "config/elasticsearch.conf"
 
wma-pdainteraction.conf
=======================
 
input {
file {
path => "C:/wms/pdainteraction.log"
#start_position => "beginning" # ignored when mode=read
codec => plain { charset => "ISO-8859-1" }
}
}
 
filter {
grok {
match => { "message" => "%{DATESTAMP:log_timestamp} \[%{DATA:java_thread}\] \[%{DATA:pda_aktion}\] %{GREEDYDATA:pda_aktion_parameter} %{DATA:aktion_timestamp} \[%{DATA:benutzer}\]" }
}
date {
match => ["aktion_timestamp", "ISO8601"]
timezone => "UTC"
}
mutate {
add_field => {
"[data_stream][type]" => "logs"
"[data_stream][dataset]" => "pdainteraction"
"[data_stream][namespace]" => "prod"
}
remove_field => ["aktion_timestamp", "log_timestamp", "message"]
}
}
 
output {
pipeline { send_to => ["es-pipeline"] }
}
 
wms-wms.conf
============
 
input {
file {
path => "C:/wms/wms.log"
#start_position => "beginning" # ignored when mode=read
mode => "read"
codec => multiline {
charset => "ISO-8859-1"
pattern => "^\d{2}:\d{2}:\d{2}\.\d{3}"
negate => true
what => "previous"
}
}
}
 
filter {
grok {
match => { "message" => "%{TIME:time} %{LOGLEVEL:loglevel} +\[%{DATA:thread}\] %{DATA:logger}\:%{NUMBER:line} +%{GREEDYDATA:message}" }
add_field => { "date" => "%{+YYYY-MM-dd}"}
add_field => { "timestamp" => "%{date} %{time}" }
overwrite => ["message"]
}
 
 
date {
target => "@timestamp"
match => ["timestamp", "YYYY-MM-dd HH:mm:ss.SSS"]
}
 
mutate {
add_field => {
"[data_stream][type]" => "logs"
"[data_stream][dataset]" => "wms"
"[data_stream][namespace]" => "prod"
}
remove_field => ["date", "time", "timestamp"]
}
}
 
output {
pipeline {
send_to => ["es-pipeline"]
 
}
}
 
elasticsearch.conf
==================
 
input {
pipeline {
address => "es-pipeline"
  }
}
 
output {
elasticsearch {
hosts => ["https://server:9200"]
ssl_enabled => true
ssl_certificate_authorities => '/path/to/ca.crt'
api_key => "REDACTED"
data_stream => "true"
  }
}

Full stack trace:

[2024-06-03T13:38:50,166][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:wms-wmspipeline, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (LoadError) load error: cabin/outputs/stdlib-logger -- java.lang.StringIndexOutOfBoundsException: String index out of range: -1", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1319)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:840)"]}


Any ideas?

Thank You
2

I now found out that indeed, setting pipeline workers to 1 was neccessary, as it is stated in the multiline codec source code. The StringIndexOutOfBounds error then goes away. But it still happens after some hours of data ingestion, that the wms-wmspipeline stops delivering new data. I'm investigating with log level debug, since on default log level there's no log entry at all when this happens.

3

Solved. Problem was a combination of using only 1 pipeline worker in combination with the multiline codec, and using 'tail' instead of 'read' in the input file mode.