Hello all,
I am running an ELK stack (Elasticsearch v. 7.1.2, Filebeat v. 7.8.0, and Logstash v. 7.16.3) and recently added the following code to a Logstash .conf template (one of many in the stack which has been successfully running in a production environment [no, don't worry, I am just editing in the sandbox, I wouldn't edit in prod]):
if [json][contextMap][Log_Reason] =~ "INBOUND_*_RESPONSE" {
# Clone log and give the clone the "type" of "low_volume"
clone {
clones => ["low_volume"]
# Target_tag is datacenter ID passed in Jenkins Job as Ansible variable
add_tag => ["{{ target_tag }}"]
}
# Remove Payload if it exists
if [json][contextMap][Payload] {
mutate {
remove_field => ["[json][contextMap][Payload]"]
}
}
}
While testing my edits, I have ran into the following error being repeated on the Logstash node repeatedly.
systemd[1]: Started logstash.
logstash[116214]: Using bundled JDK: /usr/share/logstash/jdk
logstash[116214]: OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
logstash[116214]: Sending Logstash logs to /var/log/logstash which is now configured via log4j2.properties
logstash[116214]: [2022-09-23T10:42:06,481][INFO ][logstash.runner ] Log4j configuration path used is: /etc/logstash/log4j2.properties
logstash[116214]: [2022-09-23T10:42:06,489][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"7.16.3", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
logstash[116214]: [2022-09-23T10:42:07,595][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
logstash[116214]: [2022-09-23T10:42:12,094][INFO ][org.reflections.Reflections] Reflections took 65 ms to scan 1 urls, producing 119 keys and 417 values
logstash[116214]: [2022-09-23T10:42:13,689][ERROR][logstash.filters.translate] Invalid setting for translate filter plugin:
logstash[116214]: filter {
logstash[116214]: translate {
logstash[116214]: # This setting must be a path
logstash[116214]: # File does not exist or cannot be opened /etc/logstash/cust_id_translation_map.yml
logstash[116214]: dictionary_path => "/etc/logstash/cust_id_translation_map.yml"
logstash[116214]: ...
logstash[116214]: }
logstash[116214]: }
logstash[116214]: [2022-09-23T10:42:13,696][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:119)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:86)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:383)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}
logstash[116214]: warning: thread "Converge PipelineAction::Create<main>" terminated with exception (report_on_exception is true):
logstash[116214]: LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`
logstash[116214]: create at org/logstash/execution/ConvergeResultExt.java:135
logstash[116214]: add at org/logstash/execution/ConvergeResultExt.java:60
logstash[116214]: converge_state at /usr/share/logstash/logstash-core/lib/logstash/agent.rb:396
logstash[116214]: [2022-09-23T10:42:13,704][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`"}
logstash[116214]: [2022-09-23T10:42:13,883][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle `Java::JavaLang::IllegalStateException` for `PipelineAction::Create<main>`>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in `create'", "org/logstash/execution/ConvergeResultExt.java:60:in `add'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:396:in `block in converge_state'"]}
logstash[116214]: [2022-09-23T10:42:13,894][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
logstash[116214]: org.jruby.exceptions.SystemExit: (SystemExit) exit
logstash[116214]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby-complete-9.2.20.1.jar:?]
logstash[116214]: at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby-complete-9.2.20.1.jar:?]
logstash[116214]: at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:94) ~[?:?]
systemd[1]: logstash.service: main process exited, code=exited, status=1/FAILURE
systemd[1]: Unit logstash.service entered failed state.
systemd[1]: logstash.service failed.
systemd[1]: logstash.service holdoff time over, scheduling restart.
systemd[1]: Stopped logstash.
systemd[1]: Started logstash.
While looking up causes of the block in coverage_state error message, I ran into the Elastic Stack Discussion post: Logstash Configuration Error Post (2018). I assumed that since that post was resolved from a syntax issue, that the code I added above would be the source of the errors. Looking at the full error log, I am questioning whether the Invalid setting for translate filter plugin error line could be a source of the coverage_state error as well.
If anyone could offer some help or advice I would be very grateful. Thank you for your time.
