Https://discuss.elastic.co/t/logstash-not-working-with-jdk-11-0-16/315105/5

[shivani_aggarwal (Shivani)]

Hi,
This is in continuation of my prev query that got auto-closed.

Logstash not working with jdk 11.0.16

Hi @shivani_aggarwal Ok I did a quick test I used CentoOS 7, the jdk you specified and the 7.17.3 oss logstash tar.gz not the rpm.... but it looks good... see below. I suspect their is perhaps a different issue I see here like your LS_JAVA_HOME is not pointed correctly or perhaps this or this 23:20:50 chmod: cannot access '/etc/default/logstash': No such file or directory Perhaps @Badger might have an Idea.. I am not much of a dockerfile guru for logstash I also see this CentOS 7 Test $ …

Summary:
Logstash 7.17.3 is not working with jdk >= 11.0.16

As per @stephenb 's response, I've checked LS_JAVA_HOME is properly set.
I see that the installation of just logstash and service start works fine. But the problem is seen when any logstash plugin is installed.

19:08:42  + buildah run --isolation chroot working-container -- /usr/share/logstash/bin/logstash-plugin install --no-verify logstash-input-journald
19:08:42  Using bundled JDK: /usr/share/logstash/jdk
19:08:42  [0.001s][warning][os,container] Duplicate cpuset controllers detected. Picking /sys/fs/cgroup/cpuset, skipping /sys/fs/cgroup/cpuset.
19:08:42  [0.001s][warning][os,container] Duplicate cpuset controllers detected. Picking /sys/fs/cgroup/cpuset, skipping /sys/fs/cgroup/cpuset.
19:08:42  [0.001s][warning][os,container] Duplicate cpuset controllers detected. Picking /sys/fs/cgroup/cpuset, skipping /sys/fs/cgroup/cpuset.
19:08:42  WARNING: Unknown module: org.jruby.dist specified to --add-opens
19:08:42  WARNING: Unknown module: org.jruby.dist specified to --add-opens
19:08:42  WARNING: Unknown module: org.jruby.dist specified to --add-opens
19:08:42  WARNING: Unknown module: org.jruby.dist specified to --add-opens
19:08:42  Error: Could not find or load main class [0.001s][warning][os,container]
19:08:42  Caused by: java.lang.ClassNotFoundException: [0/001s][warning][os,container]
19:08:42  subprocess exited with status 1
19:08:42  subprocess exited with status 1
19:08:42  make[1]: *** [build] Error 1
19:08:42  make[1]: Leaving directory `/home/cloud-user/logstash'
19:08:42  make: *** [build] Error 1

We have been trying different combinations and now have following updates -

1. The same sequence of steps work fine with Logstash 7.17.3 with Java 11.0.15. All plugins get installed fine. The problem is hit when we move to java 11.0.16.

2. I also tried upgrading to Logstash 8.4.3 - This does not work even with 11.0.15 and gives the exact same errors in plugin installation.

Logstash 7.17.3 + java 11.0.15 -> works
Logstash 7.17.3 + java 11.0.16 (and above) -> does not work
Logstash 8.4.3 + java 11.0.15 -> does not work
Logstash 8.4.3 + java 11.0.16(and above) -> does not work

@stephenb Is it possible for you to also try installing few plugins on top of the test you did to see if there's any similar issue seen here?

Any other pointers, inputs will be much appreciated.

Thanks!

[shivani_aggarwal (Shivani)]

@stephenb Any pointers here? I'm neither able to update logstash to 8.x nor update jdk beyond 11.0.15 due to these problems.
It would really help to figure the root cause..

[stephenb (Stephen Brown)]

@shivani_aggarwal Installed Fine for me...
I think you have something else going on.
Perhaps Something with your version of CentOS or docker, these test are on Vanilla CentOS

# OSS Versions
[sbrown@stephenb-centos-logstash ~]$ ls *.gz
logstash-oss-7.17.3-linux-x86_64.tar.gz  logstash-oss-8.4.3-linux-x86_64.tar.gz

[sbrown@stephenb-centos-logstash logstash-7.17.3]$ hostnamectl
   Static hostname: stephenb-centos-logstash
         Icon name: computer-vm
           Chassis: vm
        Machine ID: sfsfasadfsadfasdf
           Boot ID: asfdasdfasdfasdfasdfasdfsadf
    Virtualization: kvm
  Operating System: CentOS Linux 7 (Core)
       CPE OS Name: cpe:/o:centos:centos:7
            Kernel: Linux 3.10.0-1160.76.1.el7.x86_64
      Architecture: x86-64

# First 7.17.3
[sbrown@stephenb-centos-logstash logstash-7.17.3]$ uname -a
Linux stephenb-centos-logstash 3.10.0-1160.76.1.el7.x86_64 #1 SMP Wed Aug 10 16:21:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

[sbrown@stephenb-centos-logstash logstash-7.17.3]$ export LS_JAVA_HOME=/etc/alternatives/jre_11_openjdk

[sbrown@stephenb-centos-logstash logstash-7.17.3]$ $LS_JAVA_HOME/bin/java -version
openjdk version "11.0.16" 2022-07-19 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.16.0.8-1.el7_9) (build 11.0.16+8-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.16.0.8-1.el7_9) (build 11.0.16+8-LTS, mixed mode, sharing)

[sbrown@stephenb-centos-logstash logstash-7.17.3]$ ./bin/logstash --version
Using LS_JAVA_HOME defined java: /etc/alternatives/jre_11_openjdk.
WARNING: Using LS_JAVA_HOME while Logstash distribution comes with a bundled JDK.
logstash 7.17.3

[sbrown@stephenb-centos-logstash logstash-7.17.3]$  bin/logstash-plugin install --no-verify logstash-input-journald
Using LS_JAVA_HOME defined java: /etc/alternatives/jre_11_openjdk.
WARNING: Using LS_JAVA_HOME while Logstash distribution comes with a bundled JDK.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Installing logstash-input-journald
Installation successful

# Now 8.4.3
[sbrown@stephenb-centos-logstash logstash-8.4.3]$ ./bin/logstash --version
Using LS_JAVA_HOME defined java: /etc/alternatives/jre_11_openjdk.
WARNING: Logstash comes bundled with the recommended JDK(Temurin-17.0.4+8), but is overridden by the version defined in LS_JAVA_HOME. Consider clearing LS_JAVA_HOME to use the bundled JDK.
logstash 8.4.3

[sbrown@stephenb-centos-logstash logstash-8.4.3]$  bin/logstash-plugin install --no-verify logstash-input-journald
Using LS_JAVA_HOME defined java: /etc/alternatives/jre_11_openjdk.
WARNING: Logstash comes bundled with the recommended JDK(Temurin-17.0.4+8), but is overridden by the version defined in LS_JAVA_HOME. Consider clearing LS_JAVA_HOME to use the bundled JDK.
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
Installing logstash-input-journald
Installation successful

LGTM!

[shivani_aggarwal (Shivani)]

Hi,
Thanks for the response. I tried some more things, would help if you could help me understand the logstash system-install script behaviour. More details below -

Background - I build a docker image of logstash using a centos7 base image with java 11.0.17 - download and install the logstash-7.17.3-oss rpm and then install some of the plugins. I use buildah to build the image -

container=$(buildah from <base-image>)

# mount the image to host
mnt_container=$(buildah mount $container)
useradd --root $mnt_container --uid 1000 --gid 0 --home-dir /usr/share/logstash --no-create-home logstash
wget <repo>/logstash-oss-7.17.3-x86_64.rpm
cp $logstash-oss-7.17.3-x86_64.rpm $mnt_container/tmp/
buildah run --isolation chroot $container -- rpm -Uvh /tmp/logstash-oss-7.17.3-x86_64.rpm
buildah run --isolation chroot $container -- /usr/share/logstash/bin/logstash-plugin install --no-verify  logstash-input-journald 

This fails with the error shared in this post earlier. Now, removed the buildah syntax from the plugin installation cmd and that worked - but it uses the bundled jdk now and not the jdk installed in my image.

Change:

$mnt_container/usr/share/logstash/bin/logstash-plugin install --no-verify logstash-input-journald

O/p (plugin install success):

/var/lib/containers/storage/vfs/dir/3cc83f50069f15/usr/share/logstash/bin/logstash-plugin install --no-verify logstash-input-journald
10:05:59  Using bundled JDK: /var/lib/containers/storage/vfs/dir/3cc83f50069f15/usr/share/logstash/jdk
10:05:59  Warning: no jvm.options file found.
10:06:17  Installing logstash-input-journald
10:07:25  Installation successful

Any pointers on what could be the issue with using the installed jdk? JDK 11.0.17 does work fine with logstash in general - only when used through buildah isolation, this is creating an issue with jdk>11.0.15.
The rpm installation with buildah isolation also does not work as expected, but does not fail it. Plugin installation however fails and stops.

10:03:05  + buildah run --isolation chroot nano-working-container -- rpm -Uvh /tmp/logstash-oss-7.17.3-x86_64.rpm
10:03:06  warning: /tmp/logstash-oss-7.17.3-x86_64.rpm: Header V4 RSA/SHA512 
10:03:07  Preparing...                          ########################################
10:03:08  Updating / installing...
10:03:15  logstash-oss-1:7.17.3-1               ########################################
10:03:17  Using JAVA_HOME defined java: /etc/alternatives/jre_openjdk/
10:03:17  WARNING: Using JAVA_HOME while Logstash distribution comes with a bundled JDK.
10:03:17  DEPRECATION: The use of JAVA_HOME is now deprecated and will be removed starting from 8.0. Please configure LS_JAVA_HOME instead.
10:03:17  Using provided startup.options file: /etc/logstash/startup.options
10:03:18  WARNING: Unknown module: org.jruby.dist specified to --add-opens
10:03:18  WARNING: Unknown module: org.jruby.dist specified to --add-opens
10:03:18  WARNING: Unknown module: org.jruby.dist specified to --add-opens
10:03:18  WARNING: Unknown module: org.jruby.dist specified to --add-opens
10:03:18  Error: Could not find or load main class [0.001s][warning][os,container]
10:03:18  Caused by: java.lang.ClassNotFoundException: [0/001s][warning][os,container]
10:03:18  /usr/share/logstash/bin/system-install: line 88: [0.001s][warning][os,container]: command not found
10:03:18  Unable to install system startup script for Logstash.
10:03:18  chmod: cannot access '/etc/default/logstash': No such file or directory
10:03:18  warning: %post(logstash-oss-1:7.17.3-1.x86_64) scriptlet failed, exit status 1

Beyond this, we don't have any more logs. Can you help us debug why /usr/share/logstash/bin/system-install is failing when not using bundled jdk ?

[stephenb (Stephen Brown)]

Apologies I am not a docker build expert... Nor an expert on the logstash install script, perhaps someone else can help.

@shivani_aggarwal

buildah

Perhaps visit the buildah forum.

Seems to be more about building your docker images than logstash.

Have you tried just building an image with the logstash version you want the JDK you want the OS you want and then just start the container and then going into the container and see if you can add the plugin?

That might tell you whether it's the build process or the actual combination.

This might not make sense. As I said I'm not a docker expert.