Logstash-forwarder-java gives SSL handshake error with Logstash v7.16.1 and higher

Elastic Stack Logstash
1

Hi folks,

We are in the process of upgrading ELK stack from v7.11.2 to v7.17.2 to fix the log4j2 vulnerability due to JNDI lookup. Most of the our servers are on AIX so we use logstash-forwarder-java utility.

We observed that some of the servers do not connect to logstash v7.17.2 and fail with the below error:

2022-04-28 13:40:02,710 INFO Forwarder - Trying to connect to <address>
2022-04-28 13:40:02,718 ERROR Forwarder - Failed to connect to server <address> : 
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
	at com.ibm.jsse2.aq.a(aq.java:62)
	at com.ibm.jsse2.aq.h(aq.java:790)
	at com.ibm.jsse2.aq.a(aq.java:622)
	at com.ibm.jsse2.aq.startHandshake(aq.java:817)
	at info.fetter.logstashforwarder.protocol.LumberjackClient.<init>(LumberjackClient.java:92)
	at info.fetter.logstashforwarder.Forwarder.connectToBeatsServer(Forwarder.java:222)
	at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:93)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
	at com.ibm.jsse2.a.a(a.java:52)
	at com.ibm.jsse2.aq.a(aq.java:684)
	... 6 more

We identified this is due to java version installed on the leaf nodes (where logstash-forwarder is installed):
Below works:

openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)

Below does not work:

java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 8.0.5.27 - pxa6480sr5fp27-20190104_01(SR5 FP27))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20181219_405297 (JIT enabled, AOT enabled)
OpenJ9   - 3f2d574
OMR      - 109ba5b
IBM      - e2996d1)
JCL - 20190104_01 based on Oracle jdk8u191-b26

I know that logstash-forwarder-java has reached end of life, but we do not have option of using filebeat as we are on AIX.

I understand that logstash is rejecting the requests, so I bumped logging level:

curl -XPUT '0.0.0.0:9601/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
    "logger.logstash.inputs.lumberjack" : "DEBUG"
}

and also changed log.level: debug But could not find anything pointing to the SSL handshake error.

My only question is: is there any other way of debugging this from logstash end ?

Thank you.

2

The support matrix makes it clear that IBM J9 has never been supported for running logstash. Personally I liked it as a JVM, but Elastic do not support it.

3

Sorry, the java versions I mentioned are on the leaf nodes (where logstash-forwarder utility is installed).

I will edit the question accordingly.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.