Hi folks,
We are in the process of upgrading ELK stack from v7.11.2 to v7.17.2 to fix the log4j2 vulnerability due to JNDI lookup. Most of the our servers are on AIX so we use logstash-forwarder-java utility.
We observed that some of the servers do not connect to logstash v7.17.2 and fail with the below error:
2022-04-28 13:40:02,710 INFO Forwarder - Trying to connect to <address>
2022-04-28 13:40:02,718 ERROR Forwarder - Failed to connect to server <address> :
javax.net.ssl.SSLHandshakeException: Remote host closed connection during handshake
at com.ibm.jsse2.aq.a(aq.java:62)
at com.ibm.jsse2.aq.h(aq.java:790)
at com.ibm.jsse2.aq.a(aq.java:622)
at com.ibm.jsse2.aq.startHandshake(aq.java:817)
at info.fetter.logstashforwarder.protocol.LumberjackClient.<init>(LumberjackClient.java:92)
at info.fetter.logstashforwarder.Forwarder.connectToBeatsServer(Forwarder.java:222)
at info.fetter.logstashforwarder.Forwarder.main(Forwarder.java:93)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
at com.ibm.jsse2.a.a(a.java:52)
at com.ibm.jsse2.aq.a(aq.java:684)
... 6 more
We identified this is due to java version installed on the leaf nodes (where logstash-forwarder is installed):
Below works:
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (build 25.252-b09, mixed mode)
Below does not work:
java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 8.0.5.27 - pxa6480sr5fp27-20190104_01(SR5 FP27))
IBM J9 VM (build 2.9, JRE 1.8.0 Linux amd64-64-Bit Compressed References 20181219_405297 (JIT enabled, AOT enabled)
OpenJ9 - 3f2d574
OMR - 109ba5b
IBM - e2996d1)
JCL - 20190104_01 based on Oracle jdk8u191-b26
I know that logstash-forwarder-java has reached end of life, but we do not have option of using filebeat as we are on AIX.
I understand that logstash is rejecting the requests, so I bumped logging level:
curl -XPUT '0.0.0.0:9601/_node/logging?pretty' -H 'Content-Type: application/json' -d'
{
"logger.logstash.inputs.lumberjack" : "DEBUG"
}
and also changed log.level: debug But could not find anything pointing to the SSL handshake error.
My only question is: is there any other way of debugging this from logstash end ?
Thank you.