Hello everybody,
i recently set up an ELK stack and am running Logstash on a Windows host which processes input from log files and sends them to elasticsearch on another host.
I use the "collector pattern" to process two different log files with different filters and send them to the "collector" pipeline which outputs the results to elasticsearch.
So there are 3 pipelines.
When i use only the output pipeline and another one of those 2 file processing pipelines, all is fine.
When i enable both, i SOMETIMES (as in every 5th try or so), when starting Logstash, i get an error:
Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:wms-wmspipeline, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (LoadError) load error: cabin/outputs/stdlib-logger -- java.lang.StringIndexOutOfBoundsException: String index out of range: -1"
The error always happens at the wms-wms pipeline!
It is not always cabin/outputs/stdlib-logger, but whenever i comment out some stuff in the pipeline configuration, it is the same StringIndexOutOfBoundsException, but related to grok-pure or cabin/mixins/timer.
Please see the full config and strack trace here:
pipelines.yml
=============
- pipeline.id: wms-pdainteractionpipeline
path.config: "config/wms-pdainteraction.conf"
- pipeline.id: wms-wmspipeline
path.config: "config/wms-wms.conf"
- pipeline.id: es
path.config: "config/elasticsearch.conf"
wma-pdainteraction.conf
=======================
input {
file {
path => "C:/wms/pdainteraction.log"
#start_position => "beginning" # ignored when mode=read
codec => plain { charset => "ISO-8859-1" }
}
}
filter {
grok {
match => { "message" => "%{DATESTAMP:log_timestamp} \[%{DATA:java_thread}\] \[%{DATA:pda_aktion}\] %{GREEDYDATA:pda_aktion_parameter} %{DATA:aktion_timestamp} \[%{DATA:benutzer}\]" }
}
date {
match => ["aktion_timestamp", "ISO8601"]
timezone => "UTC"
}
mutate {
add_field => {
"[data_stream][type]" => "logs"
"[data_stream][dataset]" => "pdainteraction"
"[data_stream][namespace]" => "prod"
}
remove_field => ["aktion_timestamp", "log_timestamp", "message"]
}
}
output {
pipeline { send_to => ["es-pipeline"] }
}
wms-wms.conf
============
input {
file {
path => "C:/wms/wms.log"
#start_position => "beginning" # ignored when mode=read
mode => "read"
codec => multiline {
charset => "ISO-8859-1"
pattern => "^\d{2}:\d{2}:\d{2}\.\d{3}"
negate => true
what => "previous"
}
}
}
filter {
grok {
match => { "message" => "%{TIME:time} %{LOGLEVEL:loglevel} +\[%{DATA:thread}\] %{DATA:logger}\:%{NUMBER:line} +%{GREEDYDATA:message}" }
add_field => { "date" => "%{+YYYY-MM-dd}"}
add_field => { "timestamp" => "%{date} %{time}" }
overwrite => ["message"]
}
date {
target => "@timestamp"
match => ["timestamp", "YYYY-MM-dd HH:mm:ss.SSS"]
}
mutate {
add_field => {
"[data_stream][type]" => "logs"
"[data_stream][dataset]" => "wms"
"[data_stream][namespace]" => "prod"
}
remove_field => ["date", "time", "timestamp"]
}
}
output {
pipeline {
send_to => ["es-pipeline"]
}
}
elasticsearch.conf
==================
input {
pipeline {
address => "es-pipeline"
}
}
output {
elasticsearch {
hosts => ["https://server:9200"]
ssl_enabled => true
ssl_certificate_authorities => '/path/to/ca.crt'
api_key => "REDACTED"
data_stream => "true"
}
}
Full stack trace:
[2024-06-03T13:38:50,166][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:wms-wmspipeline, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (LoadError) load error: cabin/outputs/stdlib-logger -- java.lang.StringIndexOutOfBoundsException: String index out of range: -1", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:186)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:847)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1319)", "org.jruby.ir.instructions.InstanceSuperInstr.interpret(InstanceSuperInstr.java:139)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:128)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:115)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.RubyClass.newInstance(RubyClass.java:931)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:90)", "org.jruby.ir.instructions.CallBase.interpret(CallBase.java:548)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:367)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.InterpreterEngine.interpret(InterpreterEngine.java:88)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.INTERPRET_METHOD(MixedModeIRMethod.java:238)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:225)", "org.jruby.internal.runtime.methods.DynamicMethod.call(DynamicMethod.java:228)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:291)", "org.jruby.ir.interpreter.InterpreterEngine.processCall(InterpreterEngine.java:328)", "org.jruby.ir.interpreter.StartupInterpreterEngine.interpret(StartupInterpreterEngine.java:66)", "org.jruby.ir.interpreter.Interpreter.INTERPRET_BLOCK(Interpreter.java:116)", "org.jruby.runtime.MixedModeIRBlockBody.commonYieldPath(MixedModeIRBlockBody.java:136)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:66)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:144)", "org.jruby.RubyProc.call(RubyProc.java:352)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:111)", "java.base/java.lang.Thread.run(Thread.java:840)"]}
Any ideas?
Thank You