Logstash are not running properly

yasar · January 12, 2023, 9:23am

Hi team,

We are trying to get the live logs from DEV Environment for testing.. But after started the Logstash (7.16.1) service, it through a error. Please check the below error.


        at RUBY.<module:LibC>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:19) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:18) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.util.set_thread_name(/usr/share/logstash/logstash-core/lib/logstash/util.rb:35) ~[?:?]
        at RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:366) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.runner.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:291) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133) ~[?:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:93) ~[?:?]

Could someone please help us to solve this issue.

Thanks,
Yasar Arafaath A.

Rios · January 12, 2023, 10:21am

It's not visible from this part of log. Can you provide full line or even better few lines above and error.

You can try:

check does another LS is already running
is/are .conf file visible for process or service
it there .lock file in /usr/share/logstash/data/.lock
do you have permission to run
set debug mode for log and configuration in logstash.yml

config.debug: true
log.level: debug

yasar · January 12, 2023, 10:58am

Hi Rios,

Please find the below logs and errors

[2023-01-11T13:03:20,095][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-01-11T13:03:20,126][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-01-11T13:03:20,678][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
org.jruby.exceptions.LoadError: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
        at org.jruby.ext.jruby.JRubyUtilLibrary.load_ext(org/jruby/ext/jruby/JRubyUtilLibrary.java:201) ~[jruby-complete-9.2.20.1.jar:?]
        at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/ffi.rb:11) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at RUBY.<module:LibC>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:19) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:18) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.util.set_thread_name(/usr/share/logstash/logstash-core/lib/logstash/util.rb:35) ~[?:?]
        at RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:366) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.runner.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:291) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133) ~[?:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:93) ~[?:?]
[2023-01-11T13:03:52,452][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-01-11T13:03:52,466][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-01-11T13:03:52,664][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
org.jruby.exceptions.LoadError: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
        at org.jruby.ext.jruby.JRubyUtilLibrary.load_ext(org/jruby/ext/jruby/JRubyUtilLibrary.java:201) ~[jruby-complete-9.2.20.1.jar:?]
        at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/ffi.rb:11) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at RUBY.<module:LibC>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:19) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:18) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.util.set_thread_name(/usr/share/logstash/logstash-core/lib/logstash/util.rb:35) ~[?:?]
        at RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:366) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.runner.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:291) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133) ~[?:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:93) ~[?:?]
[2023-01-11T13:04:24,329][INFO ][logstash.runner          ] Log4j configuration path used is: /etc/logstash/log4j2.properties
[2023-01-11T13:04:24,342][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"7.16.1", "jruby.version"=>"jruby 9.2.20.1 (2.5.8) 2021-11-30 2a2962fbd1 OpenJDK 64-Bit Server VM 11.0.13+8 on 11.0.13+8 +indy +jit [linux-x86_64]"}
[2023-01-11T13:04:24,635][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
org.jruby.exceptions.LoadError: (LoadError) Could not load FFI Provider: (NotImplementedError) FFI not available: null
 See http://jira.codehaus.org/browse/JRUBY-4583
        at org.jruby.ext.jruby.JRubyUtilLibrary.load_ext(org/jruby/ext/jruby/JRubyUtilLibrary.java:201) ~[jruby-complete-9.2.20.1.jar:?]
        at RUBY.<main>(uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/ffi.rb:11) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at RUBY.<module:LibC>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:19) ~[?:?]
        at RUBY.<main>(/usr/share/logstash/logstash-core/lib/logstash/util/prctl.rb:18) ~[?:?]
        at org.jruby.RubyKernel.require(org/jruby/RubyKernel.java:974) ~[jruby-complete-9.2.20.1.jar:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.polyglot_minus_0_dot_3_dot_5.lib.polyglot.require(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/polyglot-0.3.5/lib/polyglot.rb:65) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.util.set_thread_name(/usr/share/logstash/logstash-core/lib/logstash/util.rb:35) ~[?:?]
        at RUBY.execute(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:366) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:68) ~[?:?]
        at usr.share.logstash.logstash_minus_core.lib.logstash.runner.run(/usr/share/logstash/logstash-core/lib/logstash/runner.rb:291) ~[?:?]
        at usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133) ~[?:?]
        at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:93) ~[?:?]

[1] No other LS is running
[2] Yes, it is
[3] We don't have the .lock dir
[4] set debug mode for log and configuration in logstash.ym
We can try that

We guess, there was some issue with below path.. could you please guide us.


usr.share.logstash.vendor.bundle.jruby.$2_dot_5_dot_0.gems.clamp_minus_1_dot_0_dot_1.lib.clamp.command.run(/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/clamp-1.0.1/lib/clamp/command.rb:133) ~[?:?]

Thanks,
Yasar Arafaath A.

Rios · January 12, 2023, 12:07pm

Check this.

yasar · January 12, 2023, 12:31pm

Hi Rios,

As checked the -Djava.io.tmpdir=/var/log/logstash/tmp directory has already having on the below path section.

cd /etc/logstash/jvm options


![image|575x218](upload://p6qHZoNkpN98x5e4J41vSdajSES.png)

Can you please suggest something to solve this issue that would be really helpful to moving on further.

Thanks,
Yasar Arafaath A

Rios · January 12, 2023, 12:32pm

Is it accessible by logstash user? Might be ownership is not OK

Others can also suggest.

Badger · January 12, 2023, 6:18pm

The common reason for this is that the java.io.tmpdir is mounted noexec. Run mount to get a listing of all the mount options in use and see if /var/log/logstash is on a filesystem mounted with that option.

yasar · January 13, 2023, 4:26am

Hi Badger,

It was not mounted i guess, can you please have a look and give any solution please.


sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3###k,nr_inodes=9###6,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/nvme0n1p2 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=31,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=1###2)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,seclabel)
/dev/mapper/vgrhel-lvhome on /home type xfs (rw,nodev,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgrhel-lvvar on /var type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgrhel-lvvarlog on /var/log type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgrhel-lvvarlogaudit on /var/log/audit type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/vgrhel-lvvartmp on /var/tmp type xfs (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,noquota)
tmpfs on /run/user/1356933677 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=7##k,mode=700,uid=###77,gid=1##3)

Thanks & Regards,
Yasar Arafaath

system · February 10, 2023, 4:27am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.