Am I missing an installation step?

Elastic Stack Logstash
1

I'm running Elasticsearch on an AIX 7.1.0.0 server, and just installed Logstash v2.1.0 today. Once I untar the installation file, I attempted to run the basic example command in the tutorial, and I'm getting a Java error.

bin/logstash -e 'input { stdin { } } output { stdout { } }'

Could not find any executable java binary. Please install java in your PATH or set JAVA_HOME.

But, Elasticsearch is running well on this server, and both PATH and JAVA_HOME are set appropriately, or so it appears?

# env PATH
/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/usr/java5/jre/bin:/usr/java5/bin
# env JAVA_HOME
/usr/java8_64

Am I overlooking something obvious?

2

I don't know AIX and we don't support it unfortunately.

What does java -version report.

3

I appreciate any insight that might surface!

# java -version
java version "1.8.0"
Java(TM) SE Runtime Environment (build pap6480sr1fp10ifix-20150723_01(SR1 FP10+IV75420))
IBM J9 VM (build 2.8, JRE 1.8.0 AIX ppc64-64 Compressed References 20150722_258693 (JIT enabled, AOT enabled)
J9VM - R28_jvm.28_20150722_1718_B258693
JIT - tr.r14.java_20150625_95081.02
GC - R28_jvm.28_20150722_1718_B258693_CMPRSS
J9CL - 20150722_258693)
JCL - 20150711_01 based on Oracle jdk8u51-b15

4

I vaguely recall that logstash (ELK stack in general) doesn't play well with IBM Java. You might need to install openjdk and point LS_JAVA_OPTS (on linux it is located in /etc/sysconfig/logstash, but I'm not sure about AIX) to it or alternatively export JAVA_HOME.

-bash-4.2# ps -ef |grep ruby
root 24328 33523 0 20:42 pts/0 00:00:00 grep --color=auto ruby
logstash 34322 1 99 14:59 pts/0 15:48:58 /bin/java -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Djava.io.tmpdir=/var/lib/logstash -Xmx4096m -Xss2048k -Djffi.boot.library.path=/opt/logstash/vendor/jruby/lib/jni -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -Djava.awt.headless=true -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/logstash/heapdump.hprof -Djava.io.tmpdir=/var/lib/logstash -Xbootclasspath/a:/opt/logstash/vendor/jruby/lib/jruby.jar -classpath : -Djruby.home=/opt/logstash/vendor/jruby -Djruby.lib=/opt/logstash/vendor/jruby/lib -Djruby.script=jruby -Djruby.shell=/bin/sh org.jruby.Main --1.9 /opt/logstash/lib/bootstrap/environment.rb logstash/runner.rb agent -f /etc/logstash/conf.d -l /var/log/logstash/logstash.log -w 8
-bash-4.2# which java
/usr/bin/java
-bash-4.2# /bin/java -version
openjdk version "1.8.0_65"
OpenJDK Runtime Environment (build 1.8.0_65-b17)
OpenJDK 64-Bit Server VM (build 25.65-b01, mixed mode)

5

Thanks -- my gut feeling was telling me that IBM Java may have had something to do with it, or an incompatibility with AIX. I went to look up OpenJDK, and it looks like they had to specially port it to AIX.

I think I might try installing the Windows version of LogStash for starters, and maybe save the headache for now of adding to the Java environment on our AIX server. I may come back and revisit the OpenJDK option though.

6

I'm running into a lot of dead ends attempting to install OpenJDK for AIX, is it as straightforward as downloading and extracting the port from http://openjdkpower.osuosl.org/OpenJDK/download/bootstrap/openjdk1.7.0-ppc-aix-port-linux-ppc64-b03.tar.bz2 ?

I've done that, but the java executable in the resulting directory still doesn't seem to want to execute (even as root with appropriate permissions).

ksh: bin/java: 0403-006 Execute permission denied.

7

Making progress with the OpenJDK option ... at least I've moved beyond the "Could not find any executable java binary" error.

# bin/logstash -e 'input { stdin { } } output { stdout { } }'
DEBUG: exec /usr/download/logstash-2.1.0/vendor/jruby/bin/jruby --1.9 -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC -J-Djava.awt.headless=true -J-XX:CMSInitiatingOccupancyFraction=75 -J-XX:+UseCMSInitiatingOccupancyOnly -J-XX:+HeapDumpOnOutOfMemoryError -J-XX:HeapDumpPath=/usr/download/logstash-2.1.0/heapdump.hprof -J-Xmx1g /usr/download/logstash-2.1.0/lib/bootstrap/environment.rb logstash/runner.rb agent -e input { stdin { } } output { stdout { } }
bash: A file or directory in the path name does not exist.

I've reviewed the logstash and logstash.lib.sh scripts, and the best I can tell, is maybe something is getting lost in the JRuby execution? Maybe I'm missing an environment setting there, and it's not able to find a required file?

I've looked around ... and still haven't found the right syntax documented to include the OpenJDK path in the LS_JAVA_OPTS environment variable.

# env
_=/usr/bin/env
LANG=en_US
LOGIN=root
DEBUG=1
CLCMD_PASSTHRU=1
PATH=/usr/download/openjdk1.7.0-ppc-aix-port-b03/bin:/usr/download/openjdk1.7.0-ppc-aix-port-b03/jre/bin:/usr/download/openjdk_8u40_b13-aix-ppc64/bin:/usr/download/openjdk_8u40_b13-aix-ppc64/jre/bin:/usr/bin:/etc:/usr/sbin:/usr/ucb:/usr/bin/X11:/sbin:/usr/java8_64/jre/bin:/usr/java8_64/bin:/usr/java5/jre/bin:/usr/java5/bin:/ibm/node/bin
LC__FASTMSG=true
LOGNAME=root
MAIL=/usr/spool/mail/root
LOCPATH=/usr/lib/nls/loc
USER=root
AUTHSTATE=compat
ES_HEAP_SIZE=10g
SHELL=/usr/bin/ksh
ODMDIR=/etc/objrepos
JAVA_HOME=/usr/download/openjdk1.7.0-ppc-aix-port-b03
HOME=/
TERM=xterm
MAILMSG=[YOU HAVE NEW MAIL]
PWD=/usr/download/logstash-2.1.0
LOGSTASH_HOME=/usr/download/logstash-2.1.0
TZ=America/New_York
A__z=! LOGNAME
NLSPATH=/usr/lib/nls/msg/%L/%N:/usr/lib/nls/msg/%L/%N.cat:/usr/lib/nls/msg/%l.%c/%N:/usr/lib/nls/msg/%l.%c/%N.cat
LD_LIBRARY_PATH=/opt/freeware/lib64

8

Can you run it with --debug

9

bin/logstash --debug
DEBUG: exec /usr/download/logstash-2.1.0/vendor/jruby/bin/jruby --1.9 -J-XX:+UseParNewGC -J-XX:+UseConcMarkSweepGC -J-Djava.awt.headless=true -J-XX:CMSInitiatingOccupancyFraction=75 -J-XX:+UseCMSInitiatingOccupancyOnly -J-XX:+HeapDumpOnOutOfMemoryError -J-XX:HeapDumpPath=/usr/download/logstash-2.1.0/heapdump.hprof -J-Xmx1g /usr/download/logstash-2.1.0/lib/bootstrap/environment.rb logstash/runner.rb agent --debug
bash: A file or directory in the path name does not exist.

Looks like the same results I get when I set DEBUG=1 in the environment variables.

10

Do you have bash?

1 Like
11

We do now ... I remedied that ...

Baby steps, but at least now LogStash starts to run -- and after a few seconds, throws up a StoreError / trustAnchors error.

bin/logstash -e 'input { stdin { } } output { stdout { } }'
io/console not supported; tty will not be manipulated
hello
OpenSSL::X509::StoreError: setting default path failed: the trustAnchors parameter must be non-empty
set_default_paths at org/jruby/ext/openssl/X509Store.java:185
(root) at /usr/download/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/jruby-openssl-0.9.12-java/lib/jopenssl/load.rb:25
require at org/jruby/RubyKernel.java:1040
(root) at /usr/download/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/jruby-openssl-0.9.12-java/lib/openssl.rb:1
require at org/jruby/RubyKernel.java:1040
(root) at /usr/download/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/jruby-openssl-0.9.12-java/lib/openssl.rb:1
require at org/jruby/RubyKernel.java:1040
(root) at /usr/download/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.0-java/lib/logstash/patches/stronger_openssl_defaults.rb:1
require at org/jruby/RubyKernel.java:1040
(root) at /usr/download/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.0-java/lib/logstash/patches/stronger_openssl_defaults.rb:2
require at org/jruby/RubyKernel.java:1040
(root) at /usr/download/logstash-2.1.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.1.0-java/lib/logstash/patches.rb:1
require at org/jruby/RubyKernel.java:1040
(root) at /usr/download/logstash-2.1.0/lib/bootstrap/environment.rb:57

12

sudo update-ca-certificates -f

13

Apparently I have a larger task at hand on this AIX server ... "update-ca-certificates" is nowhere to be found.

14

I'm used to Linux, so there might be another command to update the cert in AIX. I keep forgetting that you are using AIX.

15

I may be on to something similar on the AIX side ... /usr/bin/openssl

Reading up on it to see if I can do something similar to update-ca-certificates

16

One related question ... should there be certificate files somewhere in the logstash directory structure? I'm not finding any files named *.cer

17

No, the error you are getting is due to Java not able to find your CA. I never ran into this running on RHEL 6 and 7, so my guess is logstash looks at a specific location that doesn't exist on AIX (since it was never tested on AIX). You might be able to hack it by adding this argument to your logstash init.d file

-Djavax.net.ssl.trustStore

If this was my env, I would switch to Linux and call it a day. :slight_smile:

18

I hear ya Vincent ... and that might be in the works, but only after I hit a brick wall with getting AIX working!

19

A couple of follow-up questions ... are there any flavors of Linux that won't work with LogStash (or the ELK stack)? If we have an opportunity to put this on ZLinux, is that going to be problematic, or is it close enough to RHEL, Ubuntu, etc.? Certainly ZLinux has to be closer than AIX.

On the prior question about SSL certificates ... is there a way to just turn that off entirely when running LogStash? Just thinking about other possible workarounds.

20

Depends on which distribution of Linux you will be using. Sounds like you are an IBM shop. I believe IBM primarily ships Linux on Z with RHEL, Ubuntu and SUSE. You should be covered.

I'm not sure about bypassing the CA. If you look at the call stack, the top level call is from the bootstrap code, I doubt very much that you can bypass it. But I don't have the authoritative answer.