ElasticSearch Watcher Webhook and SSL trust issue

Elastic Stack Elasticsearch
1

I'm running v7.8.1 ELK stack on GKE.
I have setup a watcher alert which on action is supposed to POST to a URL externally using webhook to https://alertapi-uat.systems.uk.foo/api/do
However, when it tries to fire the watcher alert I get errors related to SSL:

[2021-03-15T17:20:07,432][WARN ][o.e.c.s.DiagnosticTrustManager] [es-cluster-0] failed to establish trust with server at [alertapi-uat.systems.uk.foo]; the server provided a certificate with subject name [CN=alertapi-uat,OU=Middleware,O=foo,L=S,ST=SY,C=GB] and fingerprint [5a0d26c27eb7a93580394d51e7f3dc892ecddf0e]; the certificate has subject alternative names [DNS:lwm-uat.systems.uk.foo,DNS:alertapi-uat.systems.uk.foo,DNS:servicetracker-uat.systems.uk.foo,DNS:alertcontrol-uat.systems.uk.foo,DNS:cct-tsp-uat.systems.uk.foo,DNS:alertapi-uat]; the certificate is issued by [CN=foo Issuing CA02-G2,DC=HBEU,DC=ADROOT,DC=foo]; the certificate is signed by (subject [CN=foo Issuing CA02-G2,DC=HBEU,DC=ADROOT,DC=foo] fingerprint [e68d6f77bf00aff8274ba8c21681b08e1c673ba3]) signed by (subject [CN=foo ORCA G2] fingerprint [12566c2a6c26fa2e360c74fcd656aa5b45eba04c]) which is self-issued; the [CN=foo ORCA G2] certificate is not trusted in this ssl context ([(shared)])
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException

The container does include the relevent certs (not sure all of them are required !):

$ ls -l /usr/local/share/ca-certificates
total 16
-rw-r--r-- 1 root root 1980 Mar 24 2020 foo-issuing-ca02-g2.crt
-rw-r--r-- 1 root root 2427 Mar 24 2020 foo-issuing-ca02.crt
-rw-r--r-- 1 root root 1115 Mar 24 2020 foo-orca-g2.crt
-rw-r--r-- 1 root root 1549 Mar 24 2020 foo-root-ca.crt

I have even tried changing the SSL verification mode to none but still get the same errors:

apiVersion: v1
data:
  elasticsearch.yml: |
    cluster.name: "docker-cluster"
    network.host: 0.0.0.0
    discovery.type: single-node
    http.max_content_length: 500mb
    xpack.http.ssl.verification_mode: none
    xpack.http.ssl.certificate_authorities: ["/usr/local/share/ca-certificates/foo-orca-g2.crt","/usr/local/share/ca-certificates/foo-root-ca.crt","/usr/local/share/ca-certificates/foo-issuing-ca02.crt",""/usr/local/share/ca-certificates/foo-issuing-ca02-g2.crt"]

According to the documentation, if I disable this then I should be able to make non SSL verified using the option above but that doesn't seem to work:

I have also verified I can use basic curl SSL request to the alertapi-uat URL which I believe is using SSL certs on the docker image.

However still see the same errors related to SSL, have attached the verbose set of errors for reference:

sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

    at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439) ~[?:?]
    at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306) ~[?:?]
    at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
    at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231) ~[?:?]
    at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132) ~[?:?]
    at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:90) [elasticsearch-ssl-config-7.8.1.jar:7.8.1]
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:629) [?:?]
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464) [?:?]
    at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360) [?:?]
    at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:445) [?:?]
    at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:423) [?:?]
    at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) [?:?]
    at sun.security.ssl.SSLTransport.decode(SSLTransport.java:167) [?:?]
    at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1462) [?:?]
    at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1370) [?:?]
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:437) [?:?]
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) [httpclient-4.5.10.jar:4.5.10]
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) [httpclient-4.5.10.jar:4.5.10]
    at org.elasticsearch.xpack.watcher.common.http.HttpClient.lambda$execute$1(HttpClient.java:239) [x-pack-watcher-7.8.1.jar:7.8.1]
    at java.security.AccessController.doPrivileged(AccessController.java:554) [?:?]
    at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:32) [x-pack-core-7.8.1.jar:7.8.1]
    at org.elasticsearch.xpack.watcher.common.http.HttpClient.execute(HttpClient.java:239) [x-pack-watcher-7.8.1.jar:7.8.1]
    at org.elasticsearch.xpack.watcher.actions.webhook.ExecutableWebhookAction.execute(ExecutableWebhookAction.java:42) [x-pack-watcher-7.8.1.jar:7.8.1]
    at org.elasticsearch.xpack.core.watcher.actions.ActionWrapper.execute(ActionWrapper.java:164) [x-pack-core-7.8.1.jar:7.8.1]
    at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:533) [x-pack-watcher-7.8.1.jar:7.8.1]
    at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:319) [x-pack-watcher-7.8.1.jar:7.8.1]
    at org.elasticsearch.xpack.watcher.execution.ExecutionService.lambda$executeAsync$5(ExecutionService.java:420) [x-pack-watcher-7.8.1.jar:7.8.1]
    at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:626) [x-pack-watcher-7.8.1.jar:7.8.1]
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:636) [elasticsearch-7.8.1.jar:7.8.1]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
    at java.lang.Thread.run(Thread.java:832) [?:?]

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297) ~[?:?]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434) ~[?:?]
... 41 more
[2021-03-15T17:20:07,455][ERROR][o.e.x.w.a.w.ExecutableWebhookAction] [es-cluster-0] failed to execute action [f9ac378a-e3a7-4b42-b226-8a2a97666c44/foo-alert-api-uat]
javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:325) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:268) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645) ~[?:?]
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464) ~[?:?]
at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:445) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:423) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:167) ~[?:?]
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1462) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1370) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:437) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.10.jar:4.5.10]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) ~[httpclient-4.5.10.jar:4.5.10]

2

The JVM does not use the container's certificates. It has its own certificate store.

This should fail entirely. The Elasticsearch security manager will prevent it from reading files outside of the config directory, so if you try to bring up a node with that config it should not start.

So, while that is the correct setting to use, you need to point it to a certificate that is within Elasticsearch's config dir.

3

@TimV
As per your suggestion, I copied the same 4 certificates into /user/share/elasticsearch/config and updated the config to
xpack.http.ssl.verification_mode: certificate
xpack.http.ssl.certificate_authorities: ["/user/share/elasticsearch/config/foo-orca-g2.crt","/user/share/elasticsearch/config/foo-root-ca.crt","/user/share/elasticsearch/config/foo-issuing-ca02.crt",""/user/share/elasticsearch/config/foo-issuing-ca02-g2.crt"]

However, I'm still seeing the exact same error?
Do I need to do anything further? for e.g. import into a local keystore or something?
You mentioned if I try and bring it up without the certs it shouldn't start, but that's not what's going on here as it does start. It's when the watcher action is invoked, the errors are being logged. The elasticsearch instance is still up and available though.
Also, on a separate not, previously I tried with option of "xpack.http.ssl.verification_mode: none", should this not just disable any SSL verification and allowed the connection to the url anyway as per documentation?

4

@TimV
Ignore my previous post, the issue was that my elasticsearch.yaml settings weren't being picked up so whatever change I made was being ignored.
Your suggestion worked so thanks.
One question I had about JVM certificate store, I was under the impression with JVM the certificates had to be imported into a keystore? Is elasticsearch doing some internal import with the list of certs I'm specifying or something? Just wanted to understand.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.