Logstash won't start as deamon

[elasticforme (Sachin)]

I know I have seen this issue in past and was fixed by using different ls_temp dir for logstash.

but this time it won't work.

here is error message. any idea?
this dir /s1/log/logstash is wide open

stat /s1/log/logstash/ls_temp
  File: /s1/log/logstash/ls_temp
  Size: 6               Blocks: 0          IO Block: 4096   directory
Device: 803h/2051d      Inode: 133         Links: 2
Access: (0777/drwxrwxrwx)  Uid: (  995/logstash)   Gid: (  992/logstash)

[2023-06-22T00:25:10,595][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-06-22T00:25:10,598][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"8.5.3", "jruby.version"=>"jruby 9.3.9.0 (2.6.8) 2022-10-24 537cd1f8bc OpenJDK 64-Bit Server VM 17.0.5+8 on 17.0.5+8 +jit [x86_64-linux]"}
[2023-06-22T00:25:10,600][INFO ][logstash.runner          ] JVM bootstrap flags: [-Xms15g, -Xmx15g, -Djava.io.tmpdir=/s1/log/logstash/ls_temp, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=/var/lib/elasticsearch, -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log, -Djruby.regexp.interruptible=true, -Djdk.io.File.enableADS=true, --add-exports=jdk.compiler/com.sun.tools.javac.api=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.file=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.parser=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.tree=ALL-UNNAMED, --add-exports=jdk.compiler/com.sun.tools.javac.util=ALL-UNNAMED, --add-opens=java.base/java.security=ALL-UNNAMED, --add-opens=java.base/java.io=ALL-UNNAMED, --add-opens=java.base/java.nio.channels=ALL-UNNAMED, --add-opens=java.base/sun.nio.ch=ALL-UNNAMED, --add-opens=java.management/sun.management=ALL-UNNAMED]
[2023-06-22T00:25:10,627][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
org.jruby.exceptions.LoadError: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
        at org.jruby.ext.jruby.JRubyUtilLibrary.load_ext(org/jruby/ext/jruby/JRubyUtilLibrary.java:216) ~[jruby.jar:?]
        at RUBY.<main>(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/ffi-1.15.5-java/lib/ffi.rb:11) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1017) ~[jruby.jar:?]
        at RUBY.<module:LibC>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:19) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:18) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:1017) ~[jruby.jar:?]
        at RUBY.set_thread_name(/usr/share/logstash/logstash-core/lib/logstash/util.rb:36) ~[?:?]
        at RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:354) ~[?:?]
        at RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:68) ~[?:?]
        at RUBY.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:282) ~[?:?]
        at RUBY.run(/usr/share/logstash/vendor/bundle/jruby/2.6.0/gems/clamp-1.0.1/lib/clamp/command.rb:133) ~[?:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
Caused by: org.jruby.exceptions.NotImplementedError: (NotImplementedError) FFI not available: null

[Rios]

@elasticforme

Could not load FFI Provider

A similar issue with temp directory

[elasticforme (Sachin)]

Rios, Thanks for pointing out to that link. But I had already try that.

turn out to be something in log4j2.properties file. I still don't know what it was.
this machine was clone from different logstash.
I remove logstash rpm, reinstall and it work. after that I try that old files jvm.options and log4j2.properties and it failed.

I have see few difference there. but has no time to check one by one to exactly figure out which one cause the problem.

[Rios]

Thanks for feedback.
"logstash.version"=>"8.5.3" - this is not too old version
Are you sure it's log4j2.properties? Should be there old version if you didn't remove.
Maybe are both customized. Sometimes new params has been added in a new version, however should be impact. If you figure out differences/changes in 2 files, please leave info.