Error while starting up logstash block device detection unsupported or native support failed to load

Elastic Stack Logstash
1

I have been running logstash 1.5.4, elasticsearch 1.7.2 and kibana 4.1.2 for a couple of weeks without any issues. The environment is as follows:

$ uname -a
Linux picdsr000001076 2.6.32-504.16.2.el6.x86_64 #1 SMP Tue Mar 10 17:01:00 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
JAVA_HOME=/usr/java/jdk1.7.0_79
CLASSPATH=/usr/java/jdk1.7.0_79/lib

I come in this morning as get the following error.

NotImplementedError: block device detection unsupported or native support failed to load
blockdev? at org/jruby/RubyFileTest.java:67
device? at /local/0/deploy/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/filewatch-0.6.5/lib/filewatch/helper.rb:67
_sincedb_write at /local/0/deploy/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/filewatch-0.6.5/lib/filewatch/tail.rb:232
sincedb_write at /local/0/deploy/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/filewatch-0.6.5/lib/filewatch/tail.rb:205
teardown at /local/0/deploy/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-input-file-1.0.1/lib/logstash/inputs/file.rb:157
inputworker at /local/0/deploy/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:203
synchronize at org/jruby/ext/thread/Mutex.java:149
inputworker at /local/0/deploy/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:203
start_input at /local/0/deploy/logstash-1.5.4/vendor/bundle/jruby/1.9/gems/logstash-core-1.5.4-java/lib/logstash/pipeline.rb:171

This has been working for weeks. The env files have NOT changed and I am completely stumped. I see the issue was resolved on Solaris but this is a Linux machine.

Thanks in advance.
Shailen

2

Have you seen https://github.com/elastic/logstash/issues/3127 and tried the workarounds there? I've used ln -s /lib/x86_64-linux-gnu/libcrypt.so.1 /usr/lib/x86_64-linux-gnu/libcrypt.so myself on an Ubuntu 14.04 system.

3

Thank you for the quick response. Yes. I had looked at it and did run the following test and see the library:

22:28:10/nypntsrv@picdsr000001076:/local/0/deploy/elk/logstash-1.5.4]$ strace -fo jruby.failing -e trace=file vendor/jruby/bin/jruby -e 'puts File.blockdev?("/")'
NotImplementedError: block device detection unsupported or native support failed to load
blockdev? at org/jruby/RubyFileTest.java:67
(root) at -e:1
22:28:15/nypntsrv@picdsr000001076:/local/0/deploy/elk/logstash-1.5.4]$ grep crypt.so jruby.failing
13745 open("/home/nypntsrv/perl5/perlbrew/perls/perl-5.14.2/lib/5.14.2/x86_64-linux/CORE/libcrypt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
13745 open("/usr/lib64/tls/libcrypt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
13745 open("/usr/lib64/libcrypt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
13745 open("tls/x86_64/libcrypt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
13745 open("tls/libcrypt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
13745 open("x86_64/libcrypt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
13745 open("libcrypt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
13745 open("//lib/libcrypt.so.1", O_RDONLY) = 3
13745 open("/lib64/libcrypt.so.1", O_RDONLY) = 3
13759 open("libcrypt.so.1", O_RDONLY) = -1 ENOENT (No such file or directory)
13759 open("//lib/libcrypt.so.1", O_RDONLY) = 3
13759 open("/lib64/libcrypt.so.1", O_RDONLY) = 3
13759 open("/lib/libcrypt.so.1", O_RDONLY) = 9

Is it possible that the highlighted sequence is incorrect?

My LD_LIBRARY_PATH is as follows up:

LD_LIBRARY_PATH=/usr/lib64:/home/nypntdev/lib/unixodbc/lib:/home/nypntsrv/tools/ruby-stage/lib:/home/nypntsrv/tools/openldap-stage/lib:/home/nypntsrv/tools/fontconfig-stage/lib:/home/nypntsrv/tools/pycairo-stage/lib:/home/nypntsrv/tools/cairo-stage/lib:/home/nypntsrv/tools/pixman-stage/lib:/home/nypntsrv/tools/binutils/lib:/home/nypntsrv/tools/nodejs-stage/lib:/home/nypntsrv/tools/python-stage/lib/:://lib

….and

22:31:55/nypntsrv@picdsr000001076:/usr/lib64]$ ls -ltr crypt
-rwxr-xr-x 1 root root 1965856 Mar 26 2015 libcrypto.so.1.0.1e
lrwxrwxrwx 1 root root 19 Sep 5 17:04 libcrypto.so.10 -> libcrypto.so.1.0.1e
lrwxrwxrwx 1 root root 30 Sep 5 17:05 libk5crypto.so -> ../../lib64/libk5crypto.so.3.1
lrwxrwxrwx 1 root root 25 Sep 5 17:05 libcrypt.so -> ../../lib64/libcrypt.so.1
lrwxrwxrwx 1 root root 19 Sep 5 17:06 libcrypto.so -> libcrypto.so.1.0.1e

22:32:33/nypntsrv@picdsr000001076:/usr/lib64]$ ls -ltr ../../lib64/libcrypt.so.1
lrwxrwxrwx 1 root root 16 Sep 5 17:04 ../../lib64/libcrypt.so.1 -> libcrypt-2.12.so

4

Hi Magnus,

I reverted back to libcrypt-2.5.so and logstash worked. It appears that the issue occurs with libcrypt-2.12.so.

Thanks,
Shailen