Shield 2.3.5 specifcally transport SSL/TLS failing

Elastic Stack Elasticsearch
1

Trouble figuring out exactly what is going on but here is what I am seeing:

  • shield http TLS/SSL working fine
  • ES 2.3.5
  • shield 2.3.5
  • private CA
  • shield transport failing with a message that typically is associated with uncommon CA
    - unable to find valid certification path to requested target
    - javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
    - confirmed root CA is the same via SHA signature in keystores and md5sum
    - created and configured a truststore specifically to troubleshoot , no change
    - same keystore as the http is using
    - set shield.transport.ssl: false , everything works
    - this is a tribe node to another cluster
    - curl to 9300 did end in an curl (58) nss error, while to 9200 works fine
2

Is SSL only failing on your tribe node?

3

no , all nodes for transport.ssl

between tribe and other single node clusters

The tribe node also has a second ES instance for kibana they share the JKS and it isn't working there either

4

Can you provide your full shield config in the elasticsearch.yml and the output of keytool -list -v -keystore yourkeystore.jks for two of the nodes? Maybe we can just try to get the two instances on the tribe node to communicate.

5

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 3 entries

Alias name: tribe-cert
Creation date: Jul 27, 2016
Entry type: trustedCertEntry

Owner: CN=elk-tribe.company.us, OU=IT, O=company, L=City, ST=State, C=ST
Issuer: C=US, O=company LLC, OU=companyIT, CN=company Server CA
Serial number: 5a8583f1c81654b6
Valid from: Wed Jul 27 08:21:58 MST 2016 until: Fri Jul 27 08:21:58 MST 2018
Certificate fingerprints:
MD5: E2:9B:26:00:ED:FD:66:DD:1C:6E:D5:BA:2F:08:ED:71
SHA1: 76:4E:5E:63:FB:88:93:99:EF:71:48:63:EE:AB:CC:0E:62:20:B7:D2
SHA256: 2C:A0:4C:E5:F4:E6:D4:8A:ED:68:4F:0A:7B:7D:8C:91:D8:B4:1E:43:71:0A:CF:AF:51:CD:38:21:AC:0C:F9:58
Signature algorithm name: SHA256withRSA
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
redacted ...<
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:false
PathLen: undefined
]

#3: ObjectId: 2.5.29.31 Criticality=false
CRLDistributionPoints [
[DistributionPoint:
[URIName: http://redacted]
CRLIssuer:[C=US, O=company LLC, OU=companyIT, CN=company Server CA]
]]

#4: ObjectId: 2.5.29.37 Criticality=false
ExtendedKeyUsages [
clientAuth
1.3.6.1.5.5.7.3.21
0.4.0.2231.3.0
]

#5: ObjectId: 2.5.29.46 Criticality=false
FreshestCRL [
[DistributionPoint:
[URIName: http://redacted
]]

#6: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Non_repudiation
Key_Encipherment
Data_Encipherment
]

#7: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: elk-tribe.company.us
IPAddress: 192.168.255.161
]

#8: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
redacted l...
]
]

Alias name: tribe-key
Creation date: Jul 27, 2016
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=elk-tribe.company.us, OU=IT, O=company, L=City, ST=State, C=ST
Issuer: CN=elk-tribe.company.us, OU=IT, O=company, L=City, ST=State, C=ST
Serial number: 26b233c6
Valid from: Wed Jul 27 08:23:32 MST 2016 until: Mon Jul 09 08:23:32 MST 2018
Certificate fingerprints:
MD5: B8:B9:A1:8E:C1:E0:9C:21:3D:FF:BF:6F:54:88:1C:2B
SHA1: A6:C6:28:FD:AD:9D:5F:BF:E6:8F:13:34:B8:7A:F8:58:C2:26:89:D7
SHA256: 8C:F3:3E:98:F0:94:FE:F9:01:BC:AF:49:C3:6D:DB:D2:5A:34:DF:AF:68:C0:46:69:32:81:23:72:36:83:80:70
Signature algorithm name: SHA256withRSA
Version: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
Redated
]
]

Alias name: company_ca
Creation date: Jul 27, 2016
Entry type: trustedCertEntry

Owner: C=US, O=company LLC, OU=companyIT, CN=company Server CA
Issuer: C=US, O=company LLC, OU=companyIT, CN=company Server CA
Serial number: 4694d204c830a425
Valid from: Wed Dec 17 13:36:26 MST 2014 until: Sat Dec 17 13:36:26 MST 2044
Certificate fingerprints:
MD5: F0:CB:31:05:0E:05:86:3C:FA:E3:9A:61:CE:F1:B2:9F
SHA1: 2E:92:93:A7:10:AB:00:FB:1B:DF:BC:2C:FF:AA:89:CF:B6:DD:CF:68
SHA256: CA:8F:51:1D:0C:F1:CD:71:93:2A:13:D1:DD:FE:1A:CC:42:6E:F0:C3:1C:94:8E:2D:21:ED:1A:E0:09:F7:A5:26
Signature algorithm name: SHA256withRSA
Version: 3

Extensions:

#1: ObjectId: 2.5.29.35 Criticality=false
AuthorityKeyIdentifier [
KeyIdentifier [
redacted
]
]

#2: ObjectId: 2.5.29.19 Criticality=true
BasicConstraints:[
CA:true
PathLen:2147483647
]

#3: ObjectId: 2.5.29.15 Criticality=true
KeyUsage [
DigitalSignature
Key_CertSign
Crl_Sign
]

#4: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
redacted
]

shield.audit.enabled: true
shield.ssl.keystore.path: "/etc/elasticsearch/elktribe.jks"

shield.ssl.truststore.path: "/etc/elasticsearch/elktrust.jks"

shield.ssl.truststore.password: "redacted"

shield.ssl.keystore.password: "redacted"
shield.ssl.keystore.key_password: "redacted"
shield.transport.ssl: false
shield.http.ssl: true
shield.ssl.ciphers: TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA

shield.ssl.hostname_verification.resolve_name: false

shield. authc:
realms:
file1:
type: file
order: 0
enable: true
files:
users: /etc/elasticsearch/shield/users
users_roles: /etc/elasticsearch/shield/users_roles

ldap1:

type: ldap

order: 1

enabled: false

url: 'url_to_ldap1'

ldap2:

type: ldap

order: 2

enabled: false

url: 'url_to_ldap2'

6

From the tribe config from the yml

kibana:
cluster.name: company-kibana
discovery.zen.ping.unicast.hosts: ["127.0.0.1:9309"]
discovery.zen.ping.multicast.enabled: false
shield.ssl.keystore.path: "/etc/elasticsearch/elktribe.jks"

shield.ssl.truststore.path: "/etc/elasticsearch/elktrust.jks"

shield.ssl.truststore.password: "redacted"

 shield.ssl.keystore.password: "redacted"
 shield.ssl.keystore.key_password: "redacted"
 shield.transport.ssl: false
 shield.http.ssl: true
 shield.ssl.ciphers: TLS_RSA_WITH_AES_128_CBC_SHA256,TLS_RSA_WITH_AES_128_CBC_SHA

shield.ssl.hostname_verification.resolve_name: false

7

It looks like you are using a self-signed certificate and not one signed by the CA. Is this what you intended to use?

8

It isn't a self signed certificate but a private CA is used. The output is
sanitized a bit. If you notice the issuer of the tribe-cert is the company
server ca, the car cert is the last cert of the output "Alias name:
company_ca"

jaymode Jay Modi
August 11
datadude:

Alias name: tribe-keyCreation date: Jul 27, 2016Entry type:
PrivateKeyEntryCertificate chain length: 1Certificate[1]:Owner: CN=
elk-tribe.company.us, OU=IT, O=company, L=City, ST=State, C=STIssuer: CN=
elk-tribe.company.us, OU=IT, O=company, L=City, ST=State, C=ST

It looks like you are using a self-signed certificate and not one signed
by the CA. Is this what you intended to use?

Visit Topic or reply to this email to respond.

In Reply To
datadude
August 10
Keystore type: JKS Keystore provider: SUN Your keystore contains 3
entries Alias name: tribe-cert Creation date: Jul 27, 2016 Entry type:
trustedCertEntry Owner: CN=elk-tribe.company.us, OU=IT, O=company, L=City,
ST=State, C=ST Issuer: C=US, O=company LLC, OU=companyIT, CN=company Server
CA Se…

9

Yes, the issuer of tribe-cert entry is not self signed, but the certificate for the privateKeyEntry is self-signed, unless that was part of the sanitization. If I had to guess, I think the tribe-cert is actually what should be associated with the privateKeyEntry, which represents both the private key and certificate that will be used by the node. If that was the intent, you can import the signed certificate with the alias tribe-key.

10

OK yes a different alias was used at import, I will correct this and try
again.

11

seemingly fixed the certificate misstep but it still doesn't seem the communication with transport.ssl enabled is working. I am just getting timeout errors in elasticsearch...

Tribe log

[2016-08-12 16:42:47,162][DEBUG][action.admin.indices.create] [tribe1] no known master node, scheduling a retry
[2016-08-12 16:43:17,163][DEBUG][action.admin.indices.create] [tribe1] timed out while retrying [indices:admin/create] after failure (timeout [30s])
[2016-08-12 16:43:17,163][WARN ][rest.suppressed ] path: /.kibana, params: {index=.kibana}
MasterNotDiscoveredException[null]

kibana es instance log

2016-08-12 16:43:19,422][WARN ][discovery.zen ] [kibana-only] failed to validate incoming join request from node [{tribe1/kibana}{GxyIe0YeSRKhJWWHfB2OuA}{10.136.255.161}{10.136.255.161:9301}{data=false, client=true}]

12

In the tribe log, do you see any exceptions or errors related to the joining to the kibana cluster? Does the tribe client have the same cluster name as the kibana cluster?

13

There is a "failed to send join request", see below

The tribe client is not the same as the cluster name, should it be?
The tribe client name is simply "kibana" and the kibana cluster name is "kibana-only"

[2016-08-15 08:03:13,999][DEBUG][action.admin.indices.create] [tribe1] no known master node, scheduling a retry
[2016-08-15 08:03:31,849][INFO ][discovery.zen ] [tribe1/kibana] failed to send join request to master [{kibana-only}{AjBZwaHRSNaBxPYrilWxHw}{127.0.0.1}{127.0.0.1:9309}], reason [ElasticsearchTimeoutException[Timeout waiting for ta
sk.]]
[2016-08-15 08:03:43,999][DEBUG][action.admin.indices.create] [tribe1] timed out while retrying [indices:admin/create] after failure (timeout [30s])
[2016-08-15 08:03:43,999][WARN ][rest.suppressed ] path: /.kibana, params: {index=.kibana}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:226)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
[2016-08-15 08:03:51,558][DEBUG][action.admin.indices.create] [tribe1] no known master node, scheduling a retry
[2016-08-15 08:04:21,558][DEBUG][action.admin.indices.create] [tribe1] timed out while retrying [indices:admin/create] after failure (timeout [30s])
[2016-08-15 08:04:21,558][WARN ][rest.suppressed ] path: /.kibana, params: {index=.kibana}
MasterNotDiscoveredException[null]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$5.onTimeout(TransportMasterNodeAction.java:226)
at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:236)
at org.elasticsearch.cluster.service.InternalClusterService$NotifyTimeout.run(InternalClusterService.java:804)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

14

What I mean is if you specify the correct the cluster name in your tribe settings. So in your case it would be something like:

tribe:
    kibana:
        cluster.name: 'kibana-only'
15

I do have it the way you give in your example.

I was errant in the last post the "kibana-only" is the node name.

Everything still seems to work fine with the transport.ssl: false, it is only when enable is there an issue.

The tribe config is still the same as pasted above in post 6

16

Can you stop both instances, and then capture the logs from when you start both back up and share the logs as a gist?

17

found it thanks for your feedback.

It was another SAN error, easier to fix it with a hostname change to match FQDN.