Logstash fail while running with opensearch

abhaypersistent · April 6, 2022, 1:50pm

I am getting this error.

"Using bundled JDK: C:\Users\abhay_pandey\Documents\GitHub\logsdash\logstash-8.1.2-windows-x86_64\logstash-8.1.2\jdk\bin\java.exe"

OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.

Sending Logstash logs to C:/Users/abhay_pandey/Documents/GitHub/logsdash/logstash-8.1.2-windows-x86_64/logstash-8.1.2/logs which is now configured via log4j2.properties

[2022-04-06T19:13:39,862][INFO ][logstash.runner ] Log4j configuration path used is: C:\Users\abhay_pandey\Documents\GitHub\logsdash\logstash-8.1.2-windows-x86_64\logstash-8.1.2\config\log4j2.properties

[2022-04-06T19:13:39,874][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.1.2", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.14.1+1 on 11.0.14.1+1 +indy +jit [mswin32-x86_64]"}

[2022-04-06T19:13:39,876][INFO ][logstash.runner ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -Djruby.regexp.interruptible=true, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]

[2022-04-06T19:13:39,963][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified

[2022-04-06T19:13:41,984][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}

[2022-04-06T19:13:42,434][INFO ][org.reflections.Reflections] Reflections took 72 ms to scan 1 urls, producing 120 keys and 419 values

[2022-04-06T19:13:43,204][ERROR][logstash.inputs.file ] Invalid setting for file input plugin:

input {

file {

  # This setting must be a ["beginning", "end"]

  # Expected one of ["beginning", "end"], got ["begining"]

  start_position => "begining"

  ...

}

}

[2022-04-06T19:13:43,221][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ConfigurationError) Something is wrong with your configuration.", :backtrace=>["org.logstash.config.ir.CompiledPipeline.(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:85)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:837)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1169)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1156)", "org.jruby.ir.targets.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:39)", "C_3a_.Users.abhay_pandey.Documents.GitHub.logsdash.logstash_minus_8_dot_1_dot_2_minus_windows_minus_x86_64.logstash_minus_8_dot_1_dot_2.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(C:/Users/abhay_pandey/Documents/GitHub/logsdash/logstash-8.1.2-windows-x86_64/logstash-8.1.2/logstash-core/lib/logstash/java_pipeline.rb:47)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:333)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:939)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.Users.abhay_pandey.Documents.GitHub.logsdash.logstash_minus_8_dot_1_dot_2_minus_windows_minus_x86_64.logstash_minus_8_dot_1_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(C:/Users/abhay_pandey/Documents/GitHub/logsdash/logstash-8.1.2-windows-x86_64/logstash-8.1.2/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "C_3a_.Users.abhay_pandey.Documents.GitHub.logsdash.logstash_minus_8_dot_1_dot_2_minus_windows_minus_x86_64.logstash_minus_8_dot_1_dot_2.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$VARARGS(C:/Users/abhay_pandey/Documents/GitHub/logsdash/logstash-8.1.2-windows-x86_64/logstash-8.1.2/logstash-core/lib/logstash/pipeline_action/create.rb:49)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:80)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:70)", "org.jruby.ir.targets.InvokeSite.invoke(InvokeSite.java:207)", "C_3a_.Users.abhay_pandey.Documents.GitHub.logsdash.logstash_minus_8_dot_1_dot_2_minus_windows_minus_x86_64.logstash_minus_8_dot_1_dot_2.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(C:/Users/abhay_pandey/Documents/GitHub/logsdash/logstash-8.1.2-windows-x86_64/logstash-8.1.2/logstash-core/lib/logstash/agent.rb:376)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:138)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:52)", "org.jruby.runtime.Block.call(Block.java:139)", "org.jruby.RubyProc.call(RubyProc.java:318)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:105)", "java.base/java.lang.Thread.run(Thread.java:829)"]}

warning: thread "Converge PipelineAction::Create" terminated with exception (report_on_exception is true):

LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>

      create at org/logstash/execution/ConvergeResultExt.java:135

         add at org/logstash/execution/ConvergeResultExt.java:60

converge_state at C:/Users/abhay_pandey/Documents/GitHub/logsdash/logstash-8.1.2-windows-x86_64/logstash-8.1.2/logstash-core/lib/logstash/agent.rb:389

[2022-04-06T19:13:43,228][ERROR][logstash.agent ] An exception happened when converging configuration {:exception=>LogStash::Error, :message=>"Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>"}

[2022-04-06T19:13:43,233][FATAL][logstash.runner ] An unexpected error occurred! {:error=>#<LogStash::Error: Don't know how to handle Java::JavaLang::IllegalStateException for PipelineAction::Create<main>>, :backtrace=>["org/logstash/execution/ConvergeResultExt.java:135:in create'", "org/logstash/execution/ConvergeResultExt.java:60:in add'", "C:/Users/abhay_pandey/Documents/GitHub/logsdash/logstash-8.1.2-windows-x86_64/logstash-8.1.2/logstash-core/lib/logstash/agent.rb:389:in `block in converge_state'"]}

[2022-04-06T19:13:43,242][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit

org.jruby.exceptions.SystemExit: (SystemExit) exit

    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?]

    at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]

    at C_3a_.Users.abhay_pandey.Documents.GitHub.logsdash.logstash_minus_8_dot_1_dot_2_minus_windows_minus_x86_64.logstash_minus_8_dot_1_dot_2.lib.bootstrap.environment.<main>(C:\Users\abhay_pandey\Documents\GitHub\logsdash\logstash-8.1.2-windows-x86_64\logstash-8.1.2\lib\bootstrap\environment.rb:94) ~[?:?]

My Logstash config file as follows

Sample Logstash configuration for creating a simple

Beats -> Logstash -> Elasticsearch pipeline.

input {

file {

start_position => "begining"

path => "C:\Users\abhay_pandey\Documents\json-data\sample-json.log"

sincedb_path => "C:\Users\abhay_pandey\Document\null"

}

filter {

json{

source => "message"

}

output {

Elasticsearch {

hosts => "http://localhost:9200"

index => "demo-json"

#user => "elastic"

#password => "changeme"

}

stdout {

}

system · April 6, 2022, 1:50pm

OpenSearch/OpenDistro are AWS run products and differ from the original Elasticsearch and Kibana products that Elastic builds and maintains. You may need to contact them directly for further assistance.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

leandrojmp · April 6, 2022, 2:05pm

Please, format your message using the Preformatted text button in the editor, the </> button.

But from what I was able to understand, you have a typo in your input, the value for start_position it is beginning, not begining, the error log made it clear:

  # This setting must be a ["beginning", "end"]

  # Expected one of ["beginning", "end"], got ["begining"]

Also, You won't be able to use Logstash 8.1.2 with OpenSearch if you use the elasticsearch output, you need to install the third-party plugin logstash-output-opensearch, but I'm not sure it supports Logstash 8, you will need to check with the Opensearch community.

system · May 4, 2022, 2:05pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.