Kibana is not connecting with elasticsearch shield SSL

Elastic Stack Elasticsearch
1

We are try to setup the shield ssl in local machine.

Elasticsearch version : 2.2.1
Kibana version : 4.4.1
Shield version : Latest version

We generated self signed crt, key, pem file as below:

openssl req -out CSR.csr -new -newkey rsa:2048 -nodes -keyout privateKey.key
openssl req -x509 -sha256 -nodes -days 365 -newkey rsa:2048 -keyout privateKey.key -out certificate.crt
openssl req -out CSR.csr -key privateKey.key -new
openssl x509 -x509toreq -in certificate.crt -out CSR.csr -signkey privateKey.key
openssl x509 -in certificate.crt -out certificate.pem
keytool -importcert -keystore node01.jks -file certificate.pem -alias my_ca
keytool -certreq -alias node01 -keystore node01.jks -file CSR.csr -keyalg rsa -ext san=dns:XXX.com,ip:XXXX.xxxx.xxx
keytool -importcert -keystore node01.jks -file Certificate-signed.crt -alias node01
openssl x509 -in Certificate-signed.crt -out node01-signed-noheaders.crt

and added in shield configuration.

Shield configuration :

shield.http.ssl: true
shield.transport.ssl: true
shield.ssl.keystore.key_password: XXXXX
shield.ssl.keystore.password: XXXX
shield.ssl.keystore.path: /es/config/shield/node01.jks
network.host: XX.XX.XX.XX

Kibana configuration :

elasticsearch.url: "https://XXXXX:9200"
elasticsearch.username: "username"
elasticsearch.password: "password"
elasticsearch.ssl.cert: /XXX/XXX/XXX/elasticsearchtls.crt
elasticsearch.ssl.key: /XXX/XXX/XXX/elasticsearchtls.key
elasticsearch.ssl.ca: /XXX/XXX/XXX/elasticsearch.pem
elasticsearch.ssl.verify: true

So when run the kibana below error is displayed in elasticsearch log.

log [12:24:25.512] [error][elasticsearch] Request error, retrying -- self signed certificate
log [12:24:25.622] [warning][elasticsearch] Unable to revive connection: https://XXXX:9200/
log [12:24:25.624] [warning][elasticsearch] No living connections
log [12:24:25.627] [error][status][plugin:elasticsearch] Status changed from yellow to red - Unable to connect to Elasticsearch at https://XXXXXX:9200.

After that when I change the elasticsearch.ssl.verify: false Kibana is working fine. but showing some error in elasticsearch log

ElasticsearchSecurityException[missing authentication token for REST request [/_mget?timeout=0&ignore_unavailable=true&preference=1461307913497]]

In elastic client also same issue is coming. When we use rejectUnauthorized: true then client not connecting to elasticsearch.

My question is Self signed certificate is working in elasticsearch kibana or not. Or we have to buy a commercial CA certificate ? Or We are missing anything?

2

I am not clear on how you signed the certificate from the steps you posted. That seems to be many more steps than necessary to generate a self signed certificate. Did you create your own CA first and sign the certificate using your CA? Or you are really just using a true self signed certificate?

I do not think you can use a self signed certificate with Kibana and have verification enabled.

3

Yes We are using our own CA and sign the certificate using that CA. After referring so many blogs I found that generating and signing has various way's.

We are used this link to generate certificates.
https://www.sslshopper.com/article-most-common-openssl-commands.html

Can you please guide or provide me some links to generate certificates?

4

Is the elasticsearch.ssl.ca setting point to the ca certificate?

5

Yes its pem file formatted...We used openssl x509 -in certificate.crt -out certificate.pem these command to generate pem file.

6

I apologize for the delay in responding. Is it possible for you to print out the text of the certificates you are using and share them here to help debug this issue? Usually the following command will do it:

openssl x509 -in your_pem_cert -noout -text
7

Result :
openssl x509 -i certificate.pem -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 16617559692214672490 (0xe69d6e7f184e1c6a)
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=AU, ST=Some-State, O=Internet Widgits Pty Ltd
Validity
Not Before: Apr 26 07:41:45 2016 GMT
Not After : Apr 2 07:41:45 2116 GMT
Subject: C=AU, ST=Some-State, O=Internet Widgits Pty Ltd
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
Public-Key: (1024 bit)
Modulus:
00:e8:ea:6a:54:9e:02:04:8b:9c:57:e1:06:bc:2c:
2e:a8:0f:ef:c3:91:ff:82:dd:69:13:7f:49:3d:7d:
27:7f:43:36:3e:f1:c6:c6:49:8c:52:73:5d:dc:4b:
a7:0a:d3:2d:ea:5b:c7:31:83:54:8f:37:1d:db:a8:
0b:5e:97:bf:ca:cd:cf:41:4f:15:24:60:09:0d:4f:
71:53:77:78:77:d7:95:dc:53:7e:8a:b7:c4:39:b3:
8c:ab:56:df:56:a0:07:94:fd:46:4b:1c:22:41:92:
8a:a1:37:80:b7:2d:8b:6c:7e:20:f1:cf:c7:06:7e:
9f:9c:e4:61:fa:f0:9a:7c:ad
Exponent: 65537 (0x10001)
X509v3 extensions:
X509v3 Subject Key Identifier:
69:27:3E:1B:88:6C:B6:E1:BA:BE:2A:C8:D3:52:72:0A:DF:BA:AF:F5
X509v3 Authority Key Identifier:
keyid:69:27:3E:1B:88:6C:B6:E1:BA:BE:2A:C8:D3:52:72:0A:DF:BA:AF:F5

        X509v3 Basic Constraints: 
            CA:TRUE 
Signature Algorithm: sha1WithRSAEncryption 
     9a:e2:0e:f7:c5:cc:78:65:de:18:c6:c1:dd:d5:9a:fd:f2:04: 
     4b:ad:41:85:b2:82:d8:45:83:69:da:dc:5b:67:dc:ea:ee:bb: 
     c7:23:cc:09:0f:f1:4e:54:f4:58:83:8e:8b:72:1f:ad:5a:68: 
     b1:1a:9b:17:8e:c8:31:e0:98:c5:92:3b:cb:b3:0d:dd:39:c4: 
     3e:0a:72:db:ee:98:22:19:ac:8b:50:12:9d:c7:d7:3f:5c:5b: 
     18:cf:15:c0:ab:4d:ae:3e:1f:65:f1:97:6d:df:15:ef:e1:2c: 
     da:d0:15:30:b0:9c:73:78:bf:86:a9:0d:91:6f:22:a5:82:73: 
     71:b7
8

Can you also provide the output for the certificate that your elasticsearch instance is using? You can use the keytool command:

keytool -list -v -keystore /es/config/shield/node01.jks
9

keytool -list -v -keystore elasticsearchtls.jks
Enter keystore password:

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 3 entries

Alias name: my_ca
Creation date: Apr 19, 2016
Entry type: trustedCertEntry

Owner: O=Internet Widgits Pty Ltd, ST=Some-State, C=AU
Issuer: O=Internet Widgits Pty Ltd, ST=Some-State, C=AU
Serial number: d5872e6d30e834f7
Valid from: Tue Apr 19 12:42:50 IST 2016 until: Wed Apr 19 12:42:50 IST 2017
Certificate fingerprints:
MD5: F4:29:96:8D:36:8F:7E:69:F1:46:C0:FC:3C:12:03:10
SHA1: B0:8D:F4:1C:A9:DE:6D:B5:63:63:CA:FC:C7:66:EA:0E:51:B3:17:1D
SHA256: DC:C0:0B:06:B5:34:51:43:A7:E0:4E:8D:1F:6A:1C:99:C6:FD:85:B5:4C:3C:55:4C:C0:AE:29:C0:23:DC:39:69
Signature algorithm name: SHA1withRSA
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: FE ED 55 3F CD 04 90 25 A7 91 54 90 14 0C 3B 73 ..U?...%..T...;s
0010: A8 1E 89 C2 ....
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: FE ED 55 3F CD 04 90 25 A7 91 54 90 14 0C 3B 73 ..U?...%..T...;s
0010: A8 1E 89 C2 ....
]
]

Alias name: node01
Creation date: Apr 19, 2016
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown
Issuer: CN=Unknown, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown
Serial number: 23080345
Valid from: Tue Apr 19 12:46:42 IST 2016 until: Sun Apr 01 12:46:42 IST 2018
Certificate fingerprints:
MD5: A4:B8:AF:13:BF:47:1A:B2:4C:A4:A4:28:9E:86:52:6D
SHA1: 86:BC:92:B7:26:EB:ED:8A:38:3E:79:3E:D8:BE:F6:C6:80:AE:0F:68
SHA256: B1:B2:DF:56:AF:AD:6E:81:6B:AE:AC:41:DB:DD:BA:80:EA:81:F9:B0:1B:0D:3C:FC:33:71:AE:24:87:FF:7F:CE
Signature algorithm name: SHA256withRSA
Version: 3

Extensions:

#1: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: localhost
IPAddress: 127.0.0.1
]

#2: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: 81 BA DF 3C EC D0 E1 9C A9 15 53 E7 44 21 B3 D3 ...<......S.D!..
0010: 59 C7 2F 5C Y./\
]
]

Alias name: test
Creation date: Apr 19, 2016
Entry type: trustedCertEntry

Owner: O=Internet Widgits Pty Ltd, ST=Some-State, C=AU
Issuer: O=Internet Widgits Pty Ltd, ST=Some-State, C=AU
Serial number: d5872e6d30e834f7
Valid from: Tue Apr 19 12:42:50 IST 2016 until: Wed Apr 19 12:42:50 IST 2017
Certificate fingerprints:
MD5: F4:29:96:8D:36:8F:7E:69:F1:46:C0:FC:3C:12:03:10
SHA1: B0:8D:F4:1C:A9:DE:6D:B5:63:63:CA:FC:C7:66:EA:0E:51:B3:17:1D
SHA256: DC:C0:0B:06:B5:34:51:43:A7:E0:4E:8D:1F:6A:1C:99:C6:FD:85:B5:4C:3C:55:4C:C0:AE:29:C0:23:DC:39:69
Signature algorithm name: SHA1withRSA
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
0000: FE ED 55 3F CD 04 90 25 A7 91 54 90 14 0C 3B 73 ..U?...%..T...;s
0010: A8 1E 89 C2 ....
]
]

#2: ObjectId: 2.5.29.19 Criticality=false
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: FE ED 55 3F CD 04 90 25 A7 91 54 90 14 0C 3B 73 ..U?...%..T...;s
0010: A8 1E 89 C2 ....
]
]

10

You are using a self signed certificate in your private key entry node01, which is why Kibana is complaining. The Owner and Issuer are the same in that certificate. The certificate should have the issuer C=AU, ST=Some-State, O=Internet Widgits Pty Ltd which you have said is your CA.