Could not load FFI Provider: (NotImplementedError) FFI not available: null

[rohit_dhiman]

   1. Logstash version: 8.3.2

   2. Logstash installation source: RPM

   3. Running logstash by executing below command inside directory 
      /usr/share/logstash/bin

    ./logstash -f /etc/logstash/conf.d/app-uat-vl.conf -- 
     path.settings=/etc/logstash

   4. Operating System: Red Hat Enterprise Linux 8.10 (Ootpa) 

   5. OpenJDK Runtime Environment (Red_Hat-17.0.11.0.9-3)        

Issue Description :

We are trying to run logstash using below command inside directory /usr/share/logstash/bin
./logstash -f /etc/logstash/conf.d/app-uat-vl.conf --path.settings=/etc/logstash

Because "systemctl start logstash" doesnot create any logs in the log directory

log path in logstash.yml

path.data: /var/lib/logstash
path.logs: /applog/logstash

Please find the respective logs below:

2024-07-22T15:33:39,003] [FATAL] [org.logstash.Logstash] Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null

See http://jira.codehaus.org/browse/JRUBY-4583 org.jruby.exceptions.LoadError: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null

See http://jiza.codehaus.org/browse/JRUBY-4583

at org.jruby.ext.jruby.JRubyUtilLibrary.load_ext(org/jruby/ext/jruby/JRubyUtilLibrary.java:201) ~[jruby.jar:?]

at RUBY.<main>(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/ffi-1.15.5-java/lib/ffi.rb:11) ~[?:?]

org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) -[jruby.jar:?]

at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jru

at by/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]

at RUBY.<module:LibC>(/usr/share/logstash/logstash-core/lib/logstash/util/protl.rb:19) ~[?:?]

RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:18) ~[?:?]

at ) ~[?:?]

at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby.jar:?]

at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jru

by/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?] usr.share.logstash.logstash_minus_core.lib.logstash.util.set_thread_name(/usr/share/logstash/logstash-core/lib/logstash/util.rb:36

at at RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:355) ~[?:?] at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_l_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby

/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68) ~[?:?] at usr.share.logstash.logstash_minus_core.lib.logstash.runner.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:283) ~[?:?] at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_l.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby

/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133) -[7:7] at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]

We have seen couple of community blog posts

for example :

Logstash stop working due to FFI not available: null

Logstash stopped working due to FFI not available: null . I have already provided the tmp path in Jvm.options # set the I/O temp directory #-Djava.io.tmpdir=$HOME -Djava.io.tmpdir=/home/apmuser/tmp drwxrwxr-x. 2 logstash logstash 6 Sep 16 13:25 tmp and the directory is own by logstash but still logstash stops working. any help would be appreciated, [2023-09-16T15:31:20,757][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"8.9.1", "jruby.version"=>"jruby 9.3…

Logstash fails with "FFI not available" message when starting logstash on Centos 7.9

1. Logstash version (e.g. bin/logstash --version) - 8.10.2 **2. Logstash installation source ** - RPM **3. How is Logstash being run ** - systemd JVM - tried both the bundled JVM (openjdk version "17.0.8" 2023-07-18) as well as OpenJDK 11 OS version (uname -a if on a Unix-like system): Linux - 3.10.0-1160.92.1.el7.x86_64 #1 SMP Tue Jun 20 11:48:01 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux Output of ldd --version ldd (GNU libc) 2.17 Contents of /etc/redhat-release : CentOS Linux release 7…

We got to know that this issue is occurring due to lack of permissions to some /tmp directory. The logstash user should have write permission to the /tmp directory.

We tried creating a '''logstash_tmp''' folder inside '''/app''' directory and we have also provided the permissions to logstash user to this directory

drwxr-xr-x. 2 logstash logstash 4096 Jul/22 12:57 logstash_tmp

we mentioned this file path '''/app/logstash_tmp''' inside the jvm.options file inside below property and it still didnt work

-Djava.io.tmpdir=/app/logstash_tmp

Can someone please help how can we resolve this issue ?

Thanks in Advance!

[Rios]

Welcome to the community!

Have you checked this and this?

[rohit_dhiman]

Thank You!! @Rios

Yes, I checked those posts. And this issue got resolved by performing below steps.

We removed the "nonexec" permission for /tmp directory inside /etc/fstab file and then we remounted /tmp.

And this error got resolved

But now we are getting a new error

(2024-07-22T21:47:39,491][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified 
[2024-07-22T21:47:41,337] [INFO][logstash.config.source.local.configpathloader] No config files found in path (:path=>"/etc/logstash/conf.d/app-uat-v1.conf") 
[2024-07-22721:47:41,345][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.

[2024-07-22T21:47:41,643] [INFO][logstash.agent Successfully started Logstash API endpoint {:port=>9600 , ssl enabled=>false)
[2024-07-22T21:47:41,696][INFO][logstash.runner] Logstash shut down. processing because of an error: (SystemExit) exit
[2024-07-22T21:47:41,709] [FATAL][org.logstash.Logstash Logstash stopped org.jruby.exceptions. SystemExit: (SystemExit) exit
at org.jruby RubyKernel.exit(org/jruby/RubyKernel.java:747) ~[jruby.jar:?] at org.jruby RubyKernel.exit(org/jruby/RubyKernel.java:710) ~[jruby.jar:?]

In short the error is :

No config files found in path (:path=>"/etc/logstash/conf.d/app-uat-v1.conf") 

For this as well we did not found any proper solutions. Can you please check and let us know what can be the right solution for this ?

We are trying to check what is wrong with the .conf file. Also we will try to provide all the required permissions to the logstash user.

Thanks Again!

[rohit_dhiman]

Hello All,

The above issue is resolved.

We changed the path /etc/logstash/conf.d/*.conf to /etc/logstash/conf.d/app-uat-v1.conf in the pipeline.yml file. In short we made the .conf file path absolute inside pipeline.yml file.

Issue got resolved but still there are few issues left.

Such as : we can see that the logstash service keeps restarting. Even when we try to start or stop the logstash service using systemctl the logs does not appear in the logs directory. We only get logs once we run the service using below command

./logstash -f /etc/logstash/conf.d/app-uat-vl.conf --path.settings=/etc/logstash

Thanks

[Rios]

If the LS logs are missing, that most likely means the root user overtook permissions on the log dir. Something like this should help:
chown -R logstash:logstash /applog/logstash

If there is errors with the LS service check journalctl.
journalctl -u logstash.service -n 100

[leandrojmp (Leandro Pereira)]

@rohit_dhiman

We only get logs once we run the service using below command

Did you run this command as the root user or using sudo?

You should not run logstash with root or using sudo because this can change the permission of paths used by logstash and it will not start or run correctly when you run it as a service.

What do you have in the logs when you run systemctl start logstash ?

[rohit_dhiman]

Hello @leandrojmp

Initially we ran this command using root user. Later on we asked Unix Management team of our project to grant execute permissions to the user that we are using to access the files and folders in linux Server inside directory /usr/share/logstash/bin.

Also, After installing the logstash from rpm package we tried to run logstash using systemctl command, the logstash service got started but no logs were generated. We checked all the files and folders there were no logs. No files were created inside /var/log/logstash.

Then later on we changed the log path to /applog/logstash and even in that directory the logs were not generated by systemctl start logstash. Logs were observed only after we ran the command.

Let me check if the permissions are modified or not ?

Thanks!

[leandrojmp (Leandro Pereira)]

@rohit_dhiman

Logs were observed only after we ran the command.

Which user did you run this command? You didn't say.

So, if you run systemctl start logstash it will fail? This would generate logs in the system logs wihch can be /var/log/messages or /var/log/syslog.

What are the permissions for /applog/logstash? The logstash user needs to have write permissions in this path.

[rohit_dhiman]

Hello @leandrojmp and @Rios

Thanks a lot for your quick responses and your valuable inputs. We were able to resolve the existing issues.

After granting the logstash user read, write and execute permission to /applog/logstash directory which is the log directory we were able to see the logs in that directory. Even after starting logstash using systemctl command we were able to see the logs.

After this we got one more error. Just adding few lines if someone faces such issue.

We got AccessDenied issue at path "/var/lib/logstash/.lock"

So we changed the user permission from root root to logstash logstash for .lock file and that error was also resolved.

After this using systemctl we stopped the logstash service disabled it, enabled it and started the service so that it can print logs successfully.

Now logstash has been started and we can see the logs flowing to ELK.