wfhartford
July 20, 2020, 5:57pm
1
I'm using ECK 1.0 and have configured an elasticsearch, apm server, and kibana with custom certificates. The certificates are managed by cert-manager and issued by Vault. The APM server is able to connect to the elasticsearch server, but the kibana pods are not. The kibana logs show failures to connect to elasticsearch with the following message:
Request error, retrying\nGET https://juicy-admin-es-http.juicy-admin.svc:9200/_xpack => unable to get issuer certificate
Investigation of the kibana pod shows that elasticsearch appears to be configured correctly:
elasticsearch:
hosts:
- https://juicy-admin-es-http.juicy-admin.svc:9200
password: <redacted>
ssl:
certificateAuthorities: /usr/share/kibana/config/elasticsearch-certs/ca.crt
verificationMode: certificate
username: juicy-admin-juicy-admin-kibana-user
Using curl on the kibana pod makes a successful secure connection (returns HTTP 401 because I don't supply credentials):
$ curl -s https://juicy-admin-es-http.juicy-admin.svc:9200/_xpack --cacert /usr/share/kibana/config/elasticsearch-certs/ca.crt
{"error":{"root_cause":[{"type":"security_exception","reason":"missing authentication credentials for REST request [/_xpack]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}}],"type":"security_exception","reason":"missing authentication credentials for REST request [/_xpack]","header":{"WWW-Authenticate":["Bearer realm=\"security\"","ApiKey","Basic realm=\"security\" charset=\"UTF-8\""]}},"status":401}
I've searched this forum and the ECK github issues, but nothing seems similar, and most related issues are using older versions of ECK with different issues that have been fixed in ECK 1.0.
wfhartford
July 20, 2020, 6:02pm
2
Anya_Sabo
July 20, 2020, 7:26pm
3
That error makes it seem like Kibana cannot use the contents of /usr/share/kibana/config/elasticsearch-certs/ca.crt to validate the ES certificate's chain. It's very odd that curl and APM work though. It may be worth double checking that all the intermediates and root are in the CA file. If that's the case then it may be worth us moving this over to the Kibana section as it would seem more Kibana related than ECK, and you may get more informed eyes on the topic there.
wfhartford
July 20, 2020, 8:02pm
4
The ca.crt file does not include the intermediate CA, only the root CA. The elasticsearch server provides the intermediate cert to the client when connecting (in this case via localhost because i'm using kubectl port-forward):
$ openssl s_client -connect localhost:9200 -prexit -showcerts
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 CN = juicy-admin.svc.cluster.local Intermediate CA
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = juicy-admin-es-http.juicy-admin.svc.cluster.local
verify return:1
---
Certificate chain
0 s:CN = juicy-admin-es-http.juicy-admin.svc.cluster.local
i:CN = juicy-admin.svc.cluster.local Intermediate CA
-----BEGIN CERTIFICATE-----
MIID+jCCAuKgAwIBAgIUMv02wDVbHgiAEV8DvBK2LcpBCxYwDQYJKoZIhvcNAQEL
BQAwODE2MDQGA1UEAxMtanVpY3ktYWRtaW4uc3ZjLmNsdXN0ZXIubG9jYWwgSW50
ZXJtZWRpYXRlIENBMB4XDTIwMDcxOTA5MzM1N1oXDTIwMDcyMTA5MzQyN1owPDE6
MDgGA1UEAxMxanVpY3ktYWRtaW4tZXMtaHR0cC5qdWljeS1hZG1pbi5zdmMuY2x1
c3Rlci5sb2NhbDCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAL2e8ke6
OFmoFWOKhj+EKieElIugTiAVFE/bfZrI9JwM3BpbvREe4j8QwL7S6VP0upxTmqHN
NcztYw+iCxreAKQ690GAVvnj/UyF7RdXLHu0TiLevRwPqYVQNAvWFrpt5a2Ucq0w
laDfLhDPTsRV4ImZLmzK7r/iCmWSV/z+YlSmwDq0kOeAW5RVK8iFTCwRdDx2MvbP
t6AeXCSMu7FbnDTS7aIVr3m2myYy3GxfUVilt2cldxIbAYFFPhEJGXv2ItRRqokY
JevdkJ2JO30foV/BJ8qVfGmWvDpca2XjE5ODJbMcGiujNgiK/5HkorfgFoOOxpTy
0S+StPW1rDdMIfcCAwEAAaOB9zCB9DAOBgNVHQ8BAf8EBAMCA6gwHQYDVR0lBBYw
FAYIKwYBBQUHAwEGCCsGAQUFBwMCMB0GA1UdDgQWBBRZ4m1E8dsjg7Wn0pu6Xp1G
2RCjgzAfBgNVHSMEGDAWgBQTfDt21uqaNDJl2gZ5ph+cAfN7hjCBggYDVR0RBHsw
eYIfanVpY3ktYWRtaW4tZXMtaHR0cC5qdWljeS1hZG1pboIjanVpY3ktYWRtaW4t
ZXMtaHR0cC5qdWljeS1hZG1pbi5zdmOCMWp1aWN5LWFkbWluLWVzLWh0dHAuanVp
Y3ktYWRtaW4uc3ZjLmNsdXN0ZXIubG9jYWwwDQYJKoZIhvcNAQELBQADggEBACqY
m1CLA1I+YNY7kAbieLqPk7a0pz7auXavgtLohgWw7Tw8AfGTe3PkXnIVdeCJBcHG
fq5LsK1KRpZXM1VBATXKT7ic2xtAL3fwqvSCS9wooSbujsAB3QXTgCTKGXNYCZp4
nbulE/ZbPnY0N7eGnYraW/btY38+3cQgSHJvjJ3XlOyhxF9hNI51Sii99hx5650z
R5GGvlxNxYIwb+DmrfT88aGJozXsUjaWg8TB2jMklAN9NElYLudNS/oHEdzAt1Kf
P3YxAhdwpmnSeP3SWUqgk4sS+o7s6O2dDs+ebt9SUEkiR3buxeYdta0F38X8P7rL
86C0I3QPsuXdX82MT4M=
-----END CERTIFICATE-----
1 s:CN = juicy-admin.svc.cluster.local Intermediate CA
i:CN = juicy-admin.svc.cluster.local
-----BEGIN CERTIFICATE-----
MIIEBzCCAu+gAwIBAgIUQLI2F7lXKo0G+pceLgJf0I5adsEwDQYJKoZIhvcNAQEL
BQAwKDEmMCQGA1UEAxMdanVpY3ktYWRtaW4uc3ZjLmNsdXN0ZXIubG9jYWwwHhcN
MjAwNzAzMTYwMjAzWhcNNDAwNjI4MTYwMjMzWjA4MTYwNAYDVQQDEy1qdWljeS1h
ZG1pbi5zdmMuY2x1c3Rlci5sb2NhbCBJbnRlcm1lZGlhdGUgQ0EwggEiMA0GCSqG
SIb3DQEBAQUAA4IBDwAwggEKAoIBAQDKBdDaEnMLln2MQfCRouodtq4xD1kgkgR2
L/DNmvvrVNAnj8Utc9VmN5y9XO5ko/J/KmmgkkheoI/gwpY8l1X2xSotNGOT6Yk5
sTCXXHch2ecKQZ3iHXKPFbaOdQnv6T7YrZWLXPZtl2gdiu8I3gbbt6sX/lV/RQoB
cxbXN/0LuN8Pz/MWY37grYpidNBPshQ1IRKP5moYknPV1IvfTOF/r88v/Bdbi2CS
6ua/phOv+JTAPwguHSrWQm3PEe7t7R+N0WklcvTKhTi7144jxO8vaYaAqc6fiyud
ZZc7DWogJsSM/Y8qDPo1orZsi56LPe5Oli4AOdgUGabC6krVj1GzAgMBAAGjggEX
MIIBEzAOBgNVHQ8BAf8EBAMCAQYwDwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQU
E3w7dtbqmjQyZdoGeaYfnAHze4YwHwYDVR0jBBgwFoAUCHOvE0Z2hAuPtXnzqH8V
enPpZdgwXAYIKwYBBQUHAQEEUDBOMEwGCCsGAQUFBzAChkBodHRwczovL3ZhdWx0
Lmp1aWN5LWFkbWluLnN2Yy5jbHVzdGVyLmxvY2FsL3YxL3BraV9hZG1pbl9yb290
L2NhMFIGA1UdHwRLMEkwR6BFoEOGQWh0dHBzOi8vdmF1bHQuanVpY3ktYWRtaW4u
c3ZjLmNsdXN0ZXIubG9jYWwvdjEvcGtpX2FkbWluX3Jvb3QvY3JsMA0GCSqGSIb3
DQEBCwUAA4IBAQCDXTaTA+rXPGVleoP0Ey1IZdtv5kvIT1w3aOqCZsu/E4fvQ4cs
C7MHv7Qn9PgE9Di3pm2Qxf8cCzxaUBh/KoTzg8RiN095fgOrpZtpAkvNWsEqKG/7
VgZvbGmLWbVrviJvOwBSscHnL7wbQqzWU9iF01mydxDfrp0Y5n56jDz5kynpkjV/
BSjNk7jva6V+gc5Er3UmIlvp4OV80/09HFNX7Pv9uiUmXeYi5w/+zy29hs9SECQG
Q35A/LaMUKhgwyM2kjn+u3M6FqizzlBT7UcaFwH3qZ0GNKti7qphGxLPpzq78317
2L8wpenBSGUHbiS41ApG/RhHtQ6Zdg7LMWg2
-----END CERTIFICATE-----
---
Server certificate
subject=CN = juicy-admin-es-http.juicy-admin.svc.cluster.local
issuer=CN = juicy-admin.svc.cluster.local Intermediate CA
---
...
Given that, is it necessary for Kibana to have a copy of the Intermediate? If so, is ECK capable of arranging for that?