SSL/TLS on ELK cluster

Elastic Stack Elasticsearch
1

Hi ,

My ELK cluster 2.3 is working fine. I have enabled LDAP integration and using Shield plugin.
Now I want to use SSL/TLS encryption within ELK cluster.

I have put following values on both ELK nodes in the /etc/elasticsearch/elasticsearch.yml

shield.ssl.keystore.path: /etc/elasticsearch/shield/irldxvm022.jks
shield.ssl.keystore.password: password
shield.ssl.keystore.key_password: password
shield.transport.ssl: true
shield.http.ssl: true
discovery.zen.ping.unicast.hosts: ["node1:9300", "node2:9300"]

After restarting ELK on both nodes , I am not able to login to kibana. I am using haproxy for kibana.
Do I need any other configuration for SSL/TLS encryption.

Regards,
Vinod

2

Hi ,

Can somebody please suggest pointers on this ?

Regards,
Vinod

3

Did you configure haproxy to use ssl? Also, why do you use haproxy for Kibana?

4

Hi ,

Yes my haproxy is redirecting to https. I am using haproxy because I have 2 instances of kibana running.

On master node I see following logs -

[2016-07-06 13:02:05,886][WARN ][shield.transport.netty ] [irldxvm002] received plaintext http traffic on a https channel, closing connection [id: 0x29ace348, /9.126.112.35:47520 => /9.126.112.35:9200]
[2016-07-06 13:02:07,049][WARN ][shield.transport.netty ] [irldxvm002] exception caught on transport layer [[id: 0x1ac042a5, /9.126.112.72:43568 => /9.126.112.35:9300]], closing connection
javax.net.ssl.SSLHandshakeException: no cipher suites in common

On second ELK node I see following logs -

[2016-07-06 13:02:19,113][ERROR][shield.transport.netty ] [irldxvm022] SSL/TLS handshake failed, closing channel: Received fatal alert: handshake_failure
[2016-07-06 13:02:19,114][WARN ][shield.transport.netty ] [irldxvm022] exception caught on transport layer [[id: 0x74cdd36c, /9.126.112.72:43618 :> irldxvm002.irl.in.ibm.com/9.126.112.35:9300]], closing connection
javax.net.ssl.SSLException: Received fatal alert: handshake_failure

Thanks,
Vinod

5

I see almost same output on both ELK nodes -

> [root@irldxvm002 ~]# openssl s_client -connect 9.126.112.72:9300
> CONNECTED(00000003)
> 139837074614088:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
> ---
> no peer certificate available
> ---
> No client certificate CA names sent
> ---
> SSL handshake has read 0 bytes and written 247 bytes
> ---
> New, (NONE), Cipher is (NONE)
> Secure Renegotiation IS NOT supported
> Compression: NONE
> Expansion: NONE
6

It does not look like you have imported a private key properly. Can you provide the output of:

keytool -list -v -keystore /etc/elasticsearch/shield/irldxvm022.jks
7

Here it is -
I used the command -

keytool -importcert -keystore irldxvm022.jks -file cert.crt -alias irldxvm022

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: irldxvm022
Creation date: Jul 1, 2016
Entry type: trustedCertEntry

Owner: OID.0.9.2342.19200300.100.1.3=derawat2@in.ibm.com, UID=03559L744, CN=9.126.112.72, OU=Research, O=ibm.com, L=New Delhi, ST=New Delhi, C=IN
Issuer: CN=IBM INTERNAL INTERMEDIATE CA, O=International Business Machines Corporation, C=US
Serial number: 892d
Valid from: Thu Jun 30 09:30:00 IST 2016 until: Sun Jun 30 09:29:59 IST 2019
Certificate fingerprints:
	 MD5:  0A:4A:48:56:C7:63:EC:F7:85:81:AC:D7:CA:8D:2F:26
	 SHA1: 80:DD:2A:78:57:A1:C4:8F:DF:41:5F:35:BA:53:31:7C:2A:DF:68:B1
	 SHA256: B0:B5:EF:58:E4:46:2C:09:5A:55:12:9D:C9:04:23:CB:D5:77:48:69:A5:E8:06:3C:4C:09:FB:EF:55:29:FC:A8
	 Signature algorithm name: SHA256withRSA
	 Version: 3

Extensions: 

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 06 2F DA C7 9A 1F 3F 87   51 B9 4D D0 F6 DA AE 97  ./....?.Q.M.....
0010: 46 4A 9D C8                                        FJ..
]
]

#2: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [CN=CRL71, CN=IBM INTERNAL INTERMEDIATE CA, O=International Business Machines Corporation, C=US]
, DistributionPoint:
     [URIName: http://daymvs1.pok.ibm.com:2001/PKIServ/cacerts/CRL71.crl]
]]

#3: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.5.29.32.0]
[PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.1
  qualifier: 0000: 16 6A 68 74 74 70 3A 2F   2F 77 33 2D 30 33 2E 69  .jhttp://w3-03.i
0010: 62 6D 2E 63 6F 6D 2F 74   72 61 6E 73 66 6F 72 6D  bm.com/transform
0020: 2F 73 61 73 2F 61 73 2D   77 65 62 2E 6E 73 66 2F  /sas/as-web.nsf/
0030: 43 6F 6E 74 65 6E 74 44   6F 63 73 42 79 54 69 74  ContentDocsByTit
0040: 6C 65 2F 49 6E 66 6F 72   6D 61 74 69 6F 6E 2B 54  le/Information+T
0050: 65 63 68 6E 6F 6C 6F 67   79 2B 53 65 63 75 72 69  echnology+Securi
0060: 74 79 2B 53 74 61 6E 64   61 72 64 73              ty+Standards

], PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.2
  qualifier: 0000: 30 56 1A 54 55 73 65 20   6F 66 20 74 68 69 73 20  0V.TUse of this 
0010: 63 65 72 74 69 66 69 63   61 74 65 20 73 68 6F 75  certificate shou
0020: 6C 64 20 61 64 68 65 72   65 20 74 6F 20 49 42 4D  ld adhere to IBM
0030: 20 63 6F 72 70 6F 72 61   74 65 20 73 74 61 6E 64   corporate stand
0040: 61 72 64 73 20 49 54 43   53 31 30 34 20 61 6E 64  ards ITCS104 and
0050: 20 49 54 43 53 33 30 30                             ITCS300

]]  ]
]

#4: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]

#5: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 34 0C B1 F6 A8 98 3B A6   A3 BA EC E3 9B C9 B2 BF  4.....;.........
0010: 60 A6 63 6B                                        `.ck
]
]



*******************************************
*******************************************
8

Where is the private key that belongs to that certificate and what format is it in? In order for SSL to work, there must be a privateKeyEntry in the keystore but you have only imported the certificate.

9

The private is in form of irldxvm022.key but when I try to run the importcert with same aliasI get error saying that alias already exists. So what is the proper way to do this ?

10

It sounds like you have a PEM key. Unfortunately Java's keytool does not support direct importing of such a key. I believe the following should work:

openssl pkcs12 -export -inkey irldxvm022.key -in cert.crt -name irldxvm022 -out irldxvm022.p12
keytool -delete -alias irldxvm022 -keystore irldxvm022.jks
keytool -importkeystore -srckeystore irldxvm022.p12 -srcstoretype pkcs12 -destkeystore irldxvm022.jks
11

The first command gave me this error -

unable to load certificates

12

If the command below doesn't work, I suggest you ask whoever created the certificate and key if they can give them to you as a PKCS12 file. With that the two keytool commands should allow you to import the certificate and key. I don't have much information to go off of so I am trying to help you with educated guesses.

openssl pkcs12 -export -inkey irldxvm022.key -in cert.crt -inform der -name irldxvm022 -out irldxvm022.p12
13

I do see there PKCS7b type file , will that work ?

14

Could you please look into this ?
Does ES Shield support only PKCS12 type certificates ?

15

What would you like us to look into? Did you try the last command I gave you?

Shield relies on the java keystore mechanism and the key and certificate needs to be imported into it. The java keytool documentation may list what formats are supported.

16

Yes I have tried the last command but its not working for us. As I mentioned the certificate is PKCS7 and not PKCS12.
I have also tried pkcs7 option of keytool but it did not work, I get error as -
keytool error: java.lang.Exception: Input not an X.509 certificate

Thanks,
Vinod

17

Keytool does not support importing a externally created key unless it is from another keystore. Keytool treats PKCS12 files as a keystore and can import both a certificate and a key from that format and this is the only format that can be used to import an externally created key.

Is your file in DER format or PEM format? Please check for both the certificate and key file. You can check by looking at them in a text editor and seeing if the first line contains ASCII text; if it does please share that.

PKCS7 is not a valid format for a private key, which is what I am asking about. Also, please provide error messages rather than saying

18

Here are the steps we followed to generate the certificate -

  1. Put the request using -

openssl req -nodes -newkey rsa:2048 -sha256 -keyout myserver.key -out server.csr

  1. On the CA website we got the certificates available in following formats -

DER, CRT, PKCS7b

( When I download PKCS7b file , it takes .pem extension )

Attached are the screenshots of error message

Regards,
Vinod

cert-error.JPG1057×73 14.3 KB

19

Please try the following using the pkcs7 formatted certificate.

openssl pkcs7 -in pkcs7_file.pem -out cert.crt -outform PEM
openssl pkcs12 -export -inkey myserver.key -in cert.crt -out myserver.p12
20

Thanks for the reply , attached is the screenshot

cert-error2.JPG927×211 41.8 KB