ES 7.8 X-Pack not returning whole cert chain root ca, intermediate and server cert

mankuuuuu · August 2, 2020, 7:27pm

I am trying to enable TLS on the elasticsearch http layer but not able to get all the certificates in the certificate chain.

Settings -

    xpack.security.enabled: true
    xpack.security.http.ssl.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.http.ssl.key: "/etc/elasticsearch/certs/server.key"
    xpack.security.http.ssl.verification_mode: certificate
    xpack.security.http.ssl.certificate: "/etc/elasticsearch/certs/server.crt"
    xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca_bundle.crt" ]
    xpack.security.transport.ssl.key: "/etc/elasticsearch/certs/common.key"
    xpack.security.transport.ssl.certificate: "/etc/elasticsearch/certs/common.crt"
    xpack.security.transport.ssl.verification_mode: none

ssl.key -> server private key
ssl.certificate -> server public certificate
ssl.ca -> root + intermediate + server

When I am doing openssl s_client -showcerts -host server.com -port 9200
Getting below output

Start Time: 1596395294
Timeout   : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
---

Is there any workaround or I am missing something here please guide me through it.
Thanks

TimV · August 3, 2020, 2:28am

The certificate settings in Elasticsearch are a little bit different than how most HTTP servers are configured.

Many HTTP servers have separate config serttings for the server's leaf certificate and the issuing chain. That seems to be what you expect here, but ES doesn't do that. Instead thexpack.security.http.ssl.certificate setting points to a certificate chain.
That is, you need to do something like:

cat root.crt intermediate.crt server.crt >> server-chain.crt

and then set

xpack.security.http.ssl.certificate: "/etc/elasticsearch/certs/server-chain.crt"

It looks like the xpack.security.http.ssl.certificate_authorities setting has lead you in the wrong direction. In ES this is the list of CAs to trust when SSL client authentication is enabled. That can be the CA (chain) that issued the node's HTTP certificate, but it does not need to be (and often isn't). If you do not have client authenticate enabled, then it essentially unused.

mankuuuuu · August 3, 2020, 6:19am

Thank you for your quick reply @TimV I have tried what you have mentioned in your comment but I am getting

Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL KeyManagerFactory
        at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:73) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:438) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1225) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:397) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:263) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$11(Node.java:484) ~[elasticsearch-7.8.0.jar:7.8.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:488) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
        ... 6 more
Caused by: java.security.KeyStoreException: Certificate chain is not valid
        at sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:646) ~[?:?]
        at sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:596) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:111) ~[?:?]
        at java.security.KeyStore.setKeyEntry(KeyStore.java:1167) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.CertParsingUtils.getKeyStore(CertParsingUtils.java:183) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.CertParsingUtils.keyManager(CertParsingUtils.java:174) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:71) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:438) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1225) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:397) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:263) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$11(Node.java:484) ~[elasticsearch-7.8.0.jar:7.8.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:488) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]

My root ca and intermediate is Digicert and I downloaded them from here https://www.digicert.com/kb/digicert-root-certificates.htm#roots
https://www.digicert.com/kb/digicert-root-certificates.htm#intermediates

Verified chain -

openssl verify -CAfile DigiCertHighAssuranceEVRootCA.crt.pem -untrusted DigiCertSHA2HighAssuranceServerCA.crt.pem azara.crt 
domain.crt: OK

and my order is root -> intermediates -> server.crt passing this complete bundle in the configuration throwing error of invalid certificate chain. Is anything I am doing wrong here ?

mankuuuuu · August 3, 2020, 11:18am

I have fixed this issue by changing the order of the cert chain.
It will be like leaf -> intermediate -> root.
Thanks @TimV for you help.

TimV · August 5, 2020, 4:27am

Sorry, that was my mistake - I'll blame it on Monday.

system · September 2, 2020, 4:28am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Setting up Xpack security - stuck Elasticsearch elastic-stack-security	2	474	April 14, 2020
Need help in - TLS in elastic cluster settings Elasticsearch elastic-stack-security	2	586	December 30, 2018
Elasticsearch crashes with "Certificate chain is not valid" exception Elasticsearch elastic-stack-security	2	2895	January 17, 2020
Elasticsearch TLS setup errors Elasticsearch elastic-stack-security	2	2118	June 4, 2020
Trying to figure out how to enable x-pack Elasticsearch elastic-stack-security	2	3144	January 28, 2020

ES 7.8 X-Pack not returning whole cert chain root ca, intermediate and server cert

Related topics