ES 7.8 X-Pack not returning whole cert chain root ca, intermediate and server cert

Elastic Stack Elasticsearch
1

I am trying to enable TLS on the elasticsearch http layer but not able to get all the certificates in the certificate chain.

Settings -

    xpack.security.enabled: true
    xpack.security.http.ssl.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.http.ssl.key: "/etc/elasticsearch/certs/server.key"
    xpack.security.http.ssl.verification_mode: certificate
    xpack.security.http.ssl.certificate: "/etc/elasticsearch/certs/server.crt"
    xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca_bundle.crt" ]
    xpack.security.transport.ssl.key: "/etc/elasticsearch/certs/common.key"
    xpack.security.transport.ssl.certificate: "/etc/elasticsearch/certs/common.crt"
    xpack.security.transport.ssl.verification_mode: none

ssl.key -> server private key
ssl.certificate -> server public certificate
ssl.ca -> root + intermediate + server

When I am doing openssl s_client -showcerts -host server.com -port 9200
Getting below output

Start Time: 1596395294
Timeout   : 7200 (sec)
Verify return code: 21 (unable to verify the first certificate)
---

Is there any workaround or I am missing something here please guide me through it.
Thanks

2

The certificate settings in Elasticsearch are a little bit different than how most HTTP servers are configured.

Many HTTP servers have separate config serttings for the server's leaf certificate and the issuing chain. That seems to be what you expect here, but ES doesn't do that. Instead thexpack.security.http.ssl.certificate setting points to a certificate chain.
That is, you need to do something like:

cat root.crt intermediate.crt server.crt >> server-chain.crt

and then set

xpack.security.http.ssl.certificate: "/etc/elasticsearch/certs/server-chain.crt"

It looks like the xpack.security.http.ssl.certificate_authorities setting has lead you in the wrong direction. In ES this is the list of CAs to trust when SSL client authentication is enabled. That can be the CA (chain) that issued the node's HTTP certificate, but it does not need to be (and often isn't). If you do not have client authenticate enabled, then it essentially unused.

1 Like
3

Thank you for your quick reply @TimV I have tried what you have mentioned in your comment but I am getting

Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL KeyManagerFactory
        at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:73) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:438) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1225) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:397) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:263) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$11(Node.java:484) ~[elasticsearch-7.8.0.jar:7.8.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:488) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:227) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:393) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.8.0.jar:7.8.0]
        ... 6 more
Caused by: java.security.KeyStoreException: Certificate chain is not valid
        at sun.security.pkcs12.PKCS12KeyStore.setKeyEntry(PKCS12KeyStore.java:646) ~[?:?]
        at sun.security.pkcs12.PKCS12KeyStore.engineSetKeyEntry(PKCS12KeyStore.java:596) ~[?:?]
        at sun.security.util.KeyStoreDelegator.engineSetKeyEntry(KeyStoreDelegator.java:111) ~[?:?]
        at java.security.KeyStore.setKeyEntry(KeyStore.java:1167) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.CertParsingUtils.getKeyStore(CertParsingUtils.java:183) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.CertParsingUtils.keyManager(CertParsingUtils.java:174) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.PEMKeyConfig.createKeyManager(PEMKeyConfig.java:71) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:438) ~[?:?]
        at java.util.HashMap.computeIfAbsent(HashMap.java:1225) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$5(SSLService.java:526) ~[?:?]
        at java.util.HashMap.forEach(HashMap.java:1425) ~[?:?]
        at java.util.Collections$UnmodifiableMap.forEach(Collections.java:1521) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:524) ~[?:?]
        at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createSSLService(XPackPlugin.java:397) ~[?:?]
        at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:263) ~[?:?]
        at org.elasticsearch.node.Node.lambda$new$11(Node.java:484) ~[elasticsearch-7.8.0.jar:7.8.0]
        at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
        at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1624) ~[?:?]
        at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
        at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
        at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
        at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
        at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
        at org.elasticsearch.node.Node.<init>(Node.java:488) ~[elasticsearch-7.8.0.jar:7.8.0]
        at org.elasticsearch.node.Node.<init>(Node.java:266) ~[elasticsearch-7.8.0.jar:7.8.0]

My root ca and intermediate is Digicert and I downloaded them from here https://www.digicert.com/kb/digicert-root-certificates.htm#roots
https://www.digicert.com/kb/digicert-root-certificates.htm#intermediates

Verified chain -

openssl verify -CAfile DigiCertHighAssuranceEVRootCA.crt.pem -untrusted DigiCertSHA2HighAssuranceServerCA.crt.pem azara.crt 
domain.crt: OK

and my order is root -> intermediates -> server.crt passing this complete bundle in the configuration throwing error of invalid certificate chain. Is anything I am doing wrong here ?

4

I have fixed this issue by changing the order of the cert chain.
It will be like leaf -> intermediate -> root.
Thanks @TimV for you help.

5

Sorry, that was my mistake - I'll blame it on Monday.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.