Logstash startup error

Elastic Stack Logstash
1

HI All,

I am trying to bring up Logstash on production server, getting below. Same set up working fine in UAT.

Error Message

2022-11-08T15:30:28,121][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.5.0", "jruby.version"=>"jruby 9.3.8.0 (2.6.8) 2022-09-13 98d69c9461 OpenJDK 64-Bit Server VM 17.0.4+8 on 17.0.4+8 +indy +jit [x86_64-linux]"}
[2022-11-08T15:30:28,124][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms1g, -Xmx1g, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djruby.compile.invokedynamic=true, -Djruby.jit.threshold=0, -XX:+HeapDumpOnOutOfMemoryError, -Djava.security.egd=file:/dev/urandom, -Dlog4j2.isThreadContextMapInheritable=true, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2022-11-08T15:30:28,205][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
org.jruby.exceptions.LoadError: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
        at org.jruby.ext.jruby.JRubyUtilLibrary.load_ext(org/jruby/ext/jruby/JRubyUtilLibrary.java:216) ~[jruby.jar:?]
        at RUBY.<main>(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/ffi-1.15.5-java/lib/ffi.rb:11) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1017) ~[jruby.jar:?]
        at RUBY.<module:LibC>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:19) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:18) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1017) ~[jruby.jar:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.util.set_thread_name(/usr/share/logstash/logstash-core/lib/logstash/util.rb:36) ~[?:?]
        at RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:354) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:68) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.runner.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:282) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_6_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logst

I gave the fill permission to tmp folder also but no luck.
and
i tried
" * Specify an alternate directory using the -Djava.io.tmpdir setting in the jvm.options file."

option also but no luck.

2

I think it have a problem with the version of OpenJDK.
What is the version of OpenJDK, what is your OS?
Hope my question can help to troubleshoot

3

HI @tatdat, I have resolved this issue by adding " LS_JAVA_OPTS" = /opt/tt/tmp" in startup.options file.

But now i am getting different error.

Error message is

Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"Java::JavaLang::IllegalStateException", :message=>"Unable to configure plugins: (ArgumentError) URI is not valid - host is not specified", :backtrace=>["org.logstash.config.ir.CompiledPipeline.<init>(CompiledPipeline.java:120)", "org.logstash.execution.JavaBasePipelineExt.initialize(JavaBasePipelineExt.java:85)", "org.logstash.execution.JavaBasePipelineExt$INVOKER$i$1$0$initialize.call(JavaBasePipelineExt$INVOKER$i$1$0$initialize.gen)", "org.jruby.internal.runtime.methods.JavaMethod$JavaMethodN.call(JavaMethod.java:846)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuper(IRRuntimeHelpers.java:1229)", "org.jruby.ir.runtime.IRRuntimeHelpers.instanceSuperSplatArgs(IRRuntimeHelpers.java:1202)", "org.jruby.ir.targets.indy.InstanceSuperInvokeSite.invoke(InstanceSuperInvokeSite.java:29)", "usr.share.logstash.logstash_minus_core.lib.logstash.java_pipeline.RUBY$method$initialize$0(/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)", "org.jruby.runtime.callsite.CachingCallSite.cacheAndCall(CachingCallSite.java:329)", "org.jruby.runtime.callsite.CachingCallSite.call(CachingCallSite.java:87)", "org.jruby.RubyClass.newInstance(RubyClass.java:911)", "org.jruby.RubyClass$INVOKER$i$newInstance.call(RubyClass$INVOKER$i$newInstance.gen)", "org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:208)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50)", "usr.share.logstash.logstash_minus_core.lib.logstash.pipeline_action.create.RUBY$method$execute$0$__VARARGS__(/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:49)", "org.jruby.internal.runtime.methods.CompiledIRMethod.call(CompiledIRMethod.java:139)", "org.jruby.internal.runtime.methods.MixedModeIRMethod.call(MixedModeIRMethod.java:112)", "org.jruby.ir.targets.indy.InvokeSite.invoke(InvokeSite.java:208)", "usr.share.logstash.logstash_minus_core.lib.logstash.agent.RUBY$block$converge_state$2(/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386)", "org.jruby.runtime.CompiledIRBlockBody.callDirect(CompiledIRBlockBody.java:141)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:64)", "org.jruby.runtime.IRBlockBody.call(IRBlockBody.java:58)", "org.jruby.runtime.Block.call(Block.java:143)", "org.jruby.RubyProc.call(RubyProc.java:309)", "org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:107)", "java.base/java.lang.Thread.run(Thread.java:833)"]}
[2022-11-09T11:09:32,075][INFO ][logstash.runner          ] Logstash shut down.

But the same configi file is working fine in nonproduction environment.

COnfig file is

input {
    kafka {
      bootstrap_servers => "ABC.xyz.aa:9092,ABB.xyz.aa:9092"
      topics => ["kfk.app.prod"]
      ssl_truststore_location => "/etc/logstash/certs/kafka.broker.truststore.jks"
      ssl_truststore_password => "password"
      security_protocol => "SSL"



    }
}

filter {

        json {
                source => "message"
             }
         }


output {
  elasticsearch {
    hosts => "https://BBB.xyz.aa:9200,https://AAA.xyz.aa:9200"
    index => "app-logs-%{+dd.MM.YYYY}"
    user => "elastic"
    password => "Master@123"
    cacert => "/etc/logstash/elk-stack.crt.pem"
    ssl_certificate_verification => false
    ilm_rollover_alias => "app-logs"
    ilm_policy => "a-logs"

  }

}
4

This should be an array.

hosts => [ "https://BBB.xyz.aa:9200", "https://AAA.xyz.aa:9200" ]

Change it and see if it works.

1 Like
5

Yes, that is correct; change it to array as he suggests. Everything will work as well

6

Hi @leandrojmp , Yes its working . There was some issue with certificates path laos but it did give proper error. Thank You for your quick response.

7

I was also facing similar issue then I found this is basically from "noexec" restriction on /tmp folder defined under "/etc/fstab" file. Remove "noexec" option - your Logstash will starting normally (atleast this happens for me :smile:

8

The /tmp filesystem is noexec for security reasons. Turning it off is a terrible idea.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.