Logstash with IBM JDK 1.8 and TLS connections

Elastic Stack Logstash
1

Hi,
I'm having a problem with Logstash (version 1.5.4, 2.4.0 or 5.0.0-rc1) and the IBM JDK version 1.8 where a NodeJS application can not connect using TLS. In the Logstash log file, I get a message about "no cipher suites in common". If I try the Oracle JDK 1.8, it works....

So I am in need of some help/pointers/suggestions on where to look in the files that Logstash ships to see if I can figure out why this is happening. From what I understand, the IBM and Oracle JDK's are essentially the same but I know that the version strings returned are vastly different ....

2

I don't use IBM's JDK, so I'll have to guess --

It seems like the IBM JDK, when doing the SSL/TLS handshake with your NodeJS app, is not finding any common cipher suites. Part of SSL negotiation is for each side to agree on a cipher to use, and if there are no ciphers in common. I don't know much about IBM's cryptography area, but my guess is that this is a problem with the IBM JDK.

Wireshark or a similar tool may help you diagnose this as you would be able to see the cipher suites offered by the client (nodejs) to the server (logstash). Seeing the list of ciphers offered would be helpful in diagnosing this.

3

Without additional details, my recommended workaround is to use OpenJDK or Oracle for your JRE.

4

Hi Jordan,
here are lots of details on my test setup. I have two VMs, both of which have several Logstash versions (1.5.4, 2.4.0 and 5.0.0-rc1) setup. I've "altered" the logstash.lib.sh file to include these lines :

JAVA_OPTS="$JAVA_OPTS -Djavax.net.debug=all"
JAVA_OPTS="$JAVA_OPTS -Djruby.openssl.debug=true"
JAVA_OPTS="$JAVA_OPTS -Djruby.openssl.warn=true"

which gives me some additional debug info. On one of my VMs (logstashtest hostname) I have the IBM JDK 1.8 installed :
java version "1.8.0"
Java(TM) SE Runtime Environment (build pxa6480sr3fp12-20160919_01(SR3 FP12))
IBM J9 VM (build 2.8, JRE 1.8.0 Linux amd64-64 Compressed References 20160915_318796 (JIT enabled, AOT enabled)
J9VM - R28_Java8_SR3_20160915_0912_B318796
JIT - tr.r14.java.green_20160818_122998
GC - R28_Java8_SR3_20160915_0912_B318796_CMPRSS
J9CL - 20160915_318796)
JCL - 20160914_01 based on Oracle jdk8u101-b13

On the other, I have Oracle's JDK 1.8 installed :
java version "1.8.0_111"
Java(TM) SE Runtime Environment (build 1.8.0_111-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.111-b14, mixed mode)

I run logstash from the command line with the --debug option.

5

Here's what I get when I run my NodeJS client against the Logstash server with the IBM JDK :

Client app

ibmadmin@logstashtest-0:~/temp/NodeJS$ node DJC01.js
events.js:160
throw er; // Unhandled 'error' event
^

Error: socket hang up
at TLSSocket.onHangUp (_tls_wrap.js:1092:19)
at TLSSocket.g (events.js:291:16)
at emitNone (events.js:91:20)
at TLSSocket.emit (events.js:185:7)
at endReadableNT (_stream_readable.js:974:12)
at _combinedTickCallback (internal/process/next_tick.js:74:11)
at process._tickCallback (internal/process/next_tick.js:98:9)
ibmadmin@logstashtest-0:~/temp/NodeJS$

=============================================================

The Logstash console window has the following :

startLogstash-2.4.0 --debug
IBMJSSE2 will not allow protocol SSLv3 per com.ibm.jsse2.disableSSLv3 set to TRUE or default
IBMJSSEProvider2 Build-Level: -20160616
{:ssl_version=>"TLS", :ciphers=>"ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA", :options=>50335743}
1.9.3
/home/ibmadmin/workspace/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/patches/stronger_openssl_defaults.rb:5:in (root)' /home/ibmadmin/workspace/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/patches.rb:1:inrequire'
/home/ibmadmin/workspace/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/patches.rb:1:in (root)' /home/ibmadmin/workspace/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/patches.rb:5:inrequire'
/home/ibmadmin/workspace/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/patches.rb:5:in (root)' /home/ibmadmin/workspace/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/environment.rb:1:inrequire'
/home/ibmadmin/workspace/logstash-2.4.0/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/environment.rb:1:in (root)' /home/ibmadmin/workspace/logstash-2.4.0/lib/bootstrap/environment.rb:68:inrequire'
/home/ibmadmin/workspace/logstash-2.4.0/lib/bootstrap/environment.rb:68:in `(root)'
Sending logstash logs to /home/ibmadmin/workspace/logs/logstash-2.4.0.log.

IBMJSSE2 will allow RFC 5746 renegotiation per com.ibm.jsse2.renegotiate set to none or default
IBMJSSE2 will not require renegotiation indicator during initial handshake per com.ibm.jsse2.renegotiation.indicator set to OPTIONAL or default taken
IBMJSSE2 will not perform identity checking against the peer cert check during renegotiation per com.ibm.jsse2.renegotiation.peer.cert.check set to OFF or default
IBMJSSE2 will allow client initiated renegotiation per jdk.tls.rejectClientInitiatedRenegotiation set to FALSE or default

6

darn character limits :frowning:

[main]<tcp, READ: TLSv1 Handshake, length = 131
*** ClientHello, TLSv1
RandomCookie: GMT: -1213890799 bytes = { 49, 145, 22, 20, 103, 109, 115, 39, 226, 162, 42, 238, 163, 205, 188, 17, 194, 182, 167, 216, 29, 153, 60, 216, 139, 94, 159, 194 }
Session ID: {}
Cipher Suites: [SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA, SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA, SSL_DHE_RSA_WITH_AES_256_CBC_SHA, SSL_DHE_DSS_WITH_AES_256_CBC_SHA, SSL_DH_RSA_WITH_AES_256_CBC_SHA, SSL_DH_DSS_WITH_AES_256_CBC_SHA, SSL_ECDH_RSA_WITH_AES_256_CBC_SHA, SSL_ECDH_ECDSA_WITH_AES_256_CBC_SHA, SSL_RSA_WITH_AES_256_CBC_SHA, SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA, SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA, SSL_DHE_RSA_WITH_AES_128_CBC_SHA, SSL_DHE_DSS_WITH_AES_128_CBC_SHA, SSL_DH_RSA_WITH_AES_128_CBC_SHA, SSL_DH_DSS_WITH_AES_128_CBC_SHA, SSL_ECDH_RSA_WITH_AES_128_CBC_SHA, SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA, SSL_RSA_WITH_AES_128_CBC_SHA, TLS_EMPTY_RENEGOTIATION_INFO_SCSV]
Compression Methods: { 0 }
Extension ec_point_formats, formats: [uncompressed, ansiX962_compressed_prime, ansiX962_compressed_char2]
Extension elliptic_curves, curve names: {secp256r1, secp521r1, unknown curve 28, unknown curve 27, secp384r1, unknown curve 26, secp256k1, sect571r1, sect571k1, sect409k1, sect409r1, sect283k1, sect283r1}
Unsupported extension type_35, data:
Unsupported extension type_13172, data:

[read] MD5 and SHA1 hashes: len = 131
0000: 01 00 00 7f 03 01 b8 a5 7f 11 31 91 16 14 67 6d ..........1...gm
0010: 73 27 e2 a2 2a ee a3 cd bc 11 c2 b6 a7 d8 1d 99 s...............
0020: 3c d8 8b 5e 9f c2 00 00 26 c0 14 c0 0a 00 39 00 ..............9.
0030: 38 00 37 00 36 c0 0f c0 05 00 35 c0 13 c0 09 00 8.7.6.....5.....
0040: 33 00 32 00 31 00 30 c0 0e c0 04 00 2f 00 ff 01 3.2.1.0.........
0050: 00 00 30 00 0b 00 04 03 00 01 02 00 0a 00 1c 00 ..0.............
0060: 1a 00 17 00 19 00 1c 00 1b 00 18 00 1a 00 16 00 ................
0070: 0e 00 0d 00 0b 00 0c 00 09 00 0a 00 23 00 00 33 ...............3
0080: 74 00 00 t..

JsseJCE: Using MessageDigest MD5 from provider IBMJCE version 1.8
JsseJCE: Using MessageDigest SHA from provider IBMJCE version 1.8
%% Initialized: [Session-1, SSL_NULL_WITH_NULL_NULL]
[main]<tcp, fatal error: 40: no cipher suites in common
javax.net.ssl.SSLHandshakeException: no cipher suites in common
%% Invalidated: [Session-1, SSL_NULL_WITH_NULL_NULL]
[main]<tcp, SEND TLSv1 ALERT: fatal, description = handshake_failure
[main]<tcp, WRITE: TLSv1 Alert, length = 2
[main]<tcp, fatal: engine already closed. Rethrowing javax.net.ssl.SSLHandshakeException: no cipher suites in common
javax.net.ssl.SSLHandshakeException: no cipher suites in common

7

Also, I have a simple Java app to connect to Logstash (either one) and I can specify the protocol to use, and then cycles thru a list of ciphers.

/*****************************************************************************************

  • runProgram - object entry point
  • @param String[] args commandline arguements

*****************************************************************************************/
private void
runProgram(String[] args)
throws Exception
{
String env = args[0];
String hostName = args[1];
int portNumber = Integer.parseInt(args[2]);
String protocol = args[3];

  SSLSocket sslSocket = null;
  for (String cipher : CIPHER_SUITES)
  {
     try
     {
        //***************************************************************************************
        // Open a connection
        //***************************************************************************************
        sslSocket = SSLUtils.getSSLSocket(env, hostName, portNumber, protocol);
        System.out.println("Connected ... trying cipher " + cipher);

        //***************************************************************************************
        // Change to use a single cipher
        //***************************************************************************************
        sslSocket.setEnabledCipherSuites( new String[] { cipher } );

        //***************************************************************************************
        // Send a one line message
        //***************************************************************************************
        PrintWriter out =
          new PrintWriter(sslSocket.getOutputStream(), true);

        out.println("Trying cipher " + cipher);
        out.flush();
     } 
     catch(Throwable t)
     {
        System.out.println("Throwable caught while using cipher " + cipher); 
        t.printStackTrace();
     }
     finally
     {
       //***************************************************************************************
       // Close the connection
       //***************************************************************************************
       try
       {
         sslSocket.close();
       } 
       catch(Exception e)
       {
       }
     }
  }

}

8

//***************************************************************************************
//* Data : Public/Protected/Package/Private static constants in this order
//***************************************************************************************
String[] CIPHER_SUITES = { "TLS_EMPTY_RENEGOTIATION_INFO_SCSV",
"SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384",
"SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA384",
"SSL_RSA_WITH_AES_256_CBC_SHA256",
"SSL_ECDH_ECDSA_WITH_AES_256_CBC_SHA384",
"SSL_ECDH_RSA_WITH_AES_256_CBC_SHA384",
"SSL_DHE_RSA_WITH_AES_256_CBC_SHA256",
"SSL_DHE_DSS_WITH_AES_256_CBC_SHA256",
"SSL_ECDHE_ECDSA_WITH_AES_256_CBC_SHA",
"SSL_ECDHE_RSA_WITH_AES_256_CBC_SHA",
"SSL_RSA_WITH_AES_256_CBC_SHA",
"SSL_ECDH_ECDSA_WITH_AES_256_CBC_SHA",
"SSL_ECDH_RSA_WITH_AES_256_CBC_SHA",
"SSL_DHE_RSA_WITH_AES_256_CBC_SHA",
"SSL_DHE_DSS_WITH_AES_256_CBC_SHA",
"SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
"SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
"SSL_RSA_WITH_AES_128_CBC_SHA256",
"SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA256",
"SSL_ECDH_RSA_WITH_AES_128_CBC_SHA256",
"SSL_DHE_RSA_WITH_AES_128_CBC_SHA256",
"SSL_DHE_DSS_WITH_AES_128_CBC_SHA256",
"SSL_ECDHE_ECDSA_WITH_AES_128_CBC_SHA",
"SSL_ECDHE_RSA_WITH_AES_128_CBC_SHA",
"SSL_RSA_WITH_AES_128_CBC_SHA",
"SSL_ECDH_ECDSA_WITH_AES_128_CBC_SHA",
"SSL_ECDH_RSA_WITH_AES_128_CBC_SHA",
"SSL_DHE_RSA_WITH_AES_128_CBC_SHA",
"SSL_DHE_DSS_WITH_AES_128_CBC_SHA",
"SSL_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
"SSL_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
"SSL_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
"SSL_RSA_WITH_AES_256_GCM_SHA384",
"SSL_ECDH_ECDSA_WITH_AES_256_GCM_SHA384",
"SSL_ECDH_RSA_WITH_AES_256_GCM_SHA384",
"SSL_DHE_DSS_WITH_AES_256_GCM_SHA384",
"SSL_DHE_RSA_WITH_AES_256_GCM_SHA384",
"SSL_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
"SSL_RSA_WITH_AES_128_GCM_SHA256",
"SSL_ECDH_ECDSA_WITH_AES_128_GCM_SHA256",
"SSL_ECDH_RSA_WITH_AES_128_GCM_SHA256",
"SSL_DHE_RSA_WITH_AES_128_GCM_SHA256",
"SSL_DHE_DSS_WITH_AES_128_GCM_SHA256",
"SSL_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA",
"SSL_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA",
"SSL_RSA_WITH_3DES_EDE_CBC_SHA",
"SSL_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA",
"SSL_ECDH_RSA_WITH_3DES_EDE_CBC_SHA",
"SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA",
"SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA"
};

9

When I ran that Java app against my Logstash servers, there was only 1 cipher that worked : SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA

In other words, it didn't matter if I ran that Java app against Logstash w/IBM JDK 1.8 or Logstash w/Oracle JDK 1.8 . It indicated that it could only connect with one cipher ....

So it doesn't seem to follow what the stronger_openssl_defaults.rb file indicates that are Logstash's default ciphers ...

10

Please let me know if you need anything else

11

Daniel,

IBM JDK is untested and unsupported.

You should use the JDK that is tested, supported, and you've already demonstrated works for you.

1 Like
12

Glen_Smith or anyone that knows,
If you use a supported JDK (Oracle or OpenJDK), and use say Logstash 2.4.0, then what is the list of cipher suites that is available for Logstash to use??