How to install filter plugin (Elapsed) into Logstash, which is already running as a pod in K8s?
If I run Logstash from the image docker.elastic.co/logstash/logstash:8.6.1, I can't use "Elapsed" filter, as it's not installed. So I sneak into the running pod and run "bin/logstash-plugin install logstash-filter-elapsed" which works well and write out that the filter is installed, however Logstash won't work with it - logs are full of errors right after the installation. I understand the the logstash should be restarted, however, if I delete/apply it in K8s, it start from image without installed plugin.
[ERROR] 2023-03-02 17:07:36.182 [Converge PipelineAction::Create<main>] registry - Unable to load plugin. {:type=>"filter", :name=>"elapsed"}
[ERROR] 2023-03-02 17:07:36.184 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (PluginLoadingError) Couldn't find any filter plugin named 'elapsed'. Are you sure this is correct? Trying to load the elapsed filter plugin resulted in this error: Unable to load the requested plugin named elapsed of type filter. The plugin is not installed.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.AbstractPipelineExt.initialize(AbstractPipelineExt.java:181)", "org.logstash.execution.AbstractPipelineExt$INVOKER$i$initialize.call(AbstractPipelineExt$INVOKER$i$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:846)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1229)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1202)", "org.jruby.ir.targets.indy.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:29)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:309)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base/java.lang.Thread.run(Thread.java:833)"]}
[
Simple logstash config is following:
filter {
# we need to add tags for certain messages, mandatory for elapsed plugin
if "Starting" in [message] {
mutate { add_tag => "ElapsedStart" }
}
if "Processing" in [messageTemplate] {
mutate { add_tag => "ElapsedEnd" }
}
# Evaluating time between two messages with Elapsed plugin, skipping messages without tags
if "Elapsed" in [tags] {
elapsed {
end_tag => "ElapsedEnd"
start_tag => "ElapsedStart"
unique_id_field => "fields.xy"
timeout => 1800
}
}
}