SSL/TLS on ELK cluster

vienodp · July 4, 2016, 11:30am

Hi ,

My ELK cluster 2.3 is working fine. I have enabled LDAP integration and using Shield plugin.
Now I want to use SSL/TLS encryption within ELK cluster.

I have put following values on both ELK nodes in the /etc/elasticsearch/elasticsearch.yml

shield.ssl.keystore.path: /etc/elasticsearch/shield/irldxvm022.jks
shield.ssl.keystore.password: password
shield.ssl.keystore.key_password: password
shield.transport.ssl: true
shield.http.ssl: true
discovery.zen.ping.unicast.hosts: ["node1:9300", "node2:9300"]

After restarting ELK on both nodes , I am not able to login to kibana. I am using haproxy for kibana.
Do I need any other configuration for SSL/TLS encryption.

Regards,
Vinod

vienodp · July 5, 2016, 5:50am

Hi ,

Can somebody please suggest pointers on this ?

Regards,
Vinod

jaymode · July 5, 2016, 1:07pm

Did you configure haproxy to use ssl? Also, why do you use haproxy for Kibana?

vienodp · July 6, 2016, 5:22am

Hi ,

Yes my haproxy is redirecting to https. I am using haproxy because I have 2 instances of kibana running.

On master node I see following logs -

[2016-07-06 13:02:05,886][WARN ][shield.transport.netty ] [irldxvm002] received plaintext http traffic on a https channel, closing connection [id: 0x29ace348, /9.126.112.35:47520 => /9.126.112.35:9200]
[2016-07-06 13:02:07,049][WARN ][shield.transport.netty ] [irldxvm002] exception caught on transport layer [[id: 0x1ac042a5, /9.126.112.72:43568 => /9.126.112.35:9300]], closing connection
javax.net.ssl.SSLHandshakeException: no cipher suites in common

On second ELK node I see following logs -

[2016-07-06 13:02:19,113][ERROR][shield.transport.netty ] [irldxvm022] SSL/TLS handshake failed, closing channel: Received fatal alert: handshake_failure
[2016-07-06 13:02:19,114][WARN ][shield.transport.netty ] [irldxvm022] exception caught on transport layer [[id: 0x74cdd36c, /9.126.112.72:43618 :> irldxvm002.irl.in.ibm.com/9.126.112.35:9300]], closing connection
javax.net.ssl.SSLException: Received fatal alert: handshake_failure

Thanks,
Vinod

vienodp · July 6, 2016, 8:51am

I see almost same output on both ELK nodes -

> [root@irldxvm002 ~]# openssl s_client -connect 9.126.112.72:9300
> CONNECTED(00000003)
> 139837074614088:error:140790E5:SSL routines:SSL23_WRITE:ssl handshake failure:s23_lib.c:184:
> ---
> no peer certificate available
> ---
> No client certificate CA names sent
> ---
> SSL handshake has read 0 bytes and written 247 bytes
> ---
> New, (NONE), Cipher is (NONE)
> Secure Renegotiation IS NOT supported
> Compression: NONE
> Expansion: NONE

jaymode · July 6, 2016, 11:42am

It does not look like you have imported a private key properly. Can you provide the output of:

keytool -list -v -keystore /etc/elasticsearch/shield/irldxvm022.jks

vienodp · July 6, 2016, 12:03pm

Here it is -
I used the command -

keytool -importcert -keystore irldxvm022.jks -file cert.crt -alias irldxvm022

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

Alias name: irldxvm022
Creation date: Jul 1, 2016
Entry type: trustedCertEntry

Owner: OID.0.9.2342.19200300.100.1.3=derawat2@in.ibm.com, UID=03559L744, CN=9.126.112.72, OU=Research, O=ibm.com, L=New Delhi, ST=New Delhi, C=IN
Issuer: CN=IBM INTERNAL INTERMEDIATE CA, O=International Business Machines Corporation, C=US
Serial number: 892d
Valid from: Thu Jun 30 09:30:00 IST 2016 until: Sun Jun 30 09:29:59 IST 2019
Certificate fingerprints:
	 MD5:  0A:4A:48:56:C7:63:EC:F7:85:81:AC:D7:CA:8D:2F:26
	 SHA1: 80:DD:2A:78:57:A1:C4:8F:DF:41:5F:35:BA:53:31:7C:2A:DF:68:B1
	 SHA256: B0:B5:EF:58:E4:46:2C:09:5A:55:12:9D:C9:04:23:CB:D5:77:48:69:A5:E8:06:3C:4C:09:FB:EF:55:29:FC:A8
	 Signature algorithm name: SHA256withRSA
	 Version: 3

Extensions: 

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: 06 2F DA C7 9A 1F 3F 87   51 B9 4D D0 F6 DA AE 97  ./....?.Q.M.....
0010: 46 4A 9D C8                                        FJ..
]
]

#2: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
  [DistributionPoint:
     [CN=CRL71, CN=IBM INTERNAL INTERMEDIATE CA, O=International Business Machines Corporation, C=US]
, DistributionPoint:
     [URIName: http://daymvs1.pok.ibm.com:2001/PKIServ/cacerts/CRL71.crl]
]]

#3: ObjectId: 2.5.29.32 Criticality=false
CertificatePolicies [
  [CertificatePolicyId: [2.5.29.32.0]
[PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.1
  qualifier: 0000: 16 6A 68 74 74 70 3A 2F   2F 77 33 2D 30 33 2E 69  .jhttp://w3-03.i
0010: 62 6D 2E 63 6F 6D 2F 74   72 61 6E 73 66 6F 72 6D  bm.com/transform
0020: 2F 73 61 73 2F 61 73 2D   77 65 62 2E 6E 73 66 2F  /sas/as-web.nsf/
0030: 43 6F 6E 74 65 6E 74 44   6F 63 73 42 79 54 69 74  ContentDocsByTit
0040: 6C 65 2F 49 6E 66 6F 72   6D 61 74 69 6F 6E 2B 54  le/Information+T
0050: 65 63 68 6E 6F 6C 6F 67   79 2B 53 65 63 75 72 69  echnology+Securi
0060: 74 79 2B 53 74 61 6E 64   61 72 64 73              ty+Standards

], PolicyQualifierInfo: [
  qualifierID: 1.3.6.1.5.5.7.2.2
  qualifier: 0000: 30 56 1A 54 55 73 65 20   6F 66 20 74 68 69 73 20  0V.TUse of this 
0010: 63 65 72 74 69 66 69 63   61 74 65 20 73 68 6F 75  certificate shou
0020: 6C 64 20 61 64 68 65 72   65 20 74 6F 20 49 42 4D  ld adhere to IBM
0030: 20 63 6F 72 70 6F 72 61   74 65 20 73 74 61 6E 64   corporate stand
0040: 61 72 64 73 20 49 54 43   53 31 30 34 20 61 6E 64  ards ITCS104 and
0050: 20 49 54 43 53 33 30 30                             ITCS300

]]  ]
]

#4: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
  serverAuth
  clientAuth
]

#5: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
  DigitalSignature
  Key_Encipherment
]

#6: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 34 0C B1 F6 A8 98 3B A6   A3 BA EC E3 9B C9 B2 BF  4.....;.........
0010: 60 A6 63 6B                                        `.ck
]
]



*******************************************
*******************************************

jaymode · July 6, 2016, 12:22pm

Where is the private key that belongs to that certificate and what format is it in? In order for SSL to work, there must be a privateKeyEntry in the keystore but you have only imported the certificate.

vienodp · July 7, 2016, 3:59am

The private is in form of irldxvm022.key but when I try to run the importcert with same aliasI get error saying that alias already exists. So what is the proper way to do this ?

jaymode · July 7, 2016, 12:31pm

It sounds like you have a PEM key. Unfortunately Java's keytool does not support direct importing of such a key. I believe the following should work:

openssl pkcs12 -export -inkey irldxvm022.key -in cert.crt -name irldxvm022 -out irldxvm022.p12
keytool -delete -alias irldxvm022 -keystore irldxvm022.jks
keytool -importkeystore -srckeystore irldxvm022.p12 -srcstoretype pkcs12 -destkeystore irldxvm022.jks

vienodp · July 8, 2016, 4:05am

The first command gave me this error -

unable to load certificates

jaymode · July 8, 2016, 10:40am

If the command below doesn't work, I suggest you ask whoever created the certificate and key if they can give them to you as a PKCS12 file. With that the two keytool commands should allow you to import the certificate and key. I don't have much information to go off of so I am trying to help you with educated guesses.

openssl pkcs12 -export -inkey irldxvm022.key -in cert.crt -inform der -name irldxvm022 -out irldxvm022.p12

vienodp · July 8, 2016, 11:24am

I do see there PKCS7b type file , will that work ?

vienodp · July 11, 2016, 6:07am

Could you please look into this ?
Does ES Shield support only PKCS12 type certificates ?

jaymode · July 11, 2016, 11:16am

What would you like us to look into? Did you try the last command I gave you?

Shield relies on the java keystore mechanism and the key and certificate needs to be imported into it. The java keytool documentation may list what formats are supported.

vienodp · July 11, 2016, 11:34am

Yes I have tried the last command but its not working for us. As I mentioned the certificate is PKCS7 and not PKCS12.
I have also tried pkcs7 option of keytool but it did not work, I get error as -
keytool error: java.lang.Exception: Input not an X.509 certificate

Thanks,
Vinod

jaymode · July 11, 2016, 11:56am

Keytool does not support importing a externally created key unless it is from another keystore. Keytool treats PKCS12 files as a keystore and can import both a certificate and a key from that format and this is the only format that can be used to import an externally created key.

Is your file in DER format or PEM format? Please check for both the certificate and key file. You can check by looking at them in a text editor and seeing if the first line contains ASCII text; if it does please share that.

PKCS7 is not a valid format for a private key, which is what I am asking about. Also, please provide error messages rather than saying

vienodp · July 11, 2016, 12:28pm

Here are the steps we followed to generate the certificate -

Put the request using -

openssl req -nodes -newkey rsa:2048 -sha256 -keyout myserver.key -out server.csr

On the CA website we got the certificates available in following formats -

DER, CRT, PKCS7b

( When I download PKCS7b file , it takes .pem extension )

Attached are the screenshots of error message

Regards,
Vinod

jaymode · July 11, 2016, 1:37pm

Please try the following using the pkcs7 formatted certificate.

openssl pkcs7 -in pkcs7_file.pem -out cert.crt -outform PEM
openssl pkcs12 -export -inkey myserver.key -in cert.crt -out myserver.p12

vienodp · July 12, 2016, 3:26am

Thanks for the reply , attached is the screenshot

Topic		Replies	Views
LDAP integrated ELK with Shield Elasticsearch elastic-stack-security	26	5202	July 6, 2017
ELK Stack + TLS + Headache Kibana elastic-stack-security	15	2123	September 17, 2021
Securing ELK Elasticsearch	2	306	November 20, 2020
ELK cluster with x-pack kibana is not working with https Kibana	4	802	June 12, 2017
SSL Error in Kibana logs Kibana	15	1803	October 14, 2020

SSL/TLS on ELK cluster

Related topics