Input Watcher problem with https

Elastic Stack Kibana
1

Dear team,

we have defined a http input watcher to query status of cluster.
ELS cluster is secured with node certificates and https communication.
When watcher is executed we got this:

[2021-09-12T11:16:48,633][ERROR][o.e.x.w.i.h.ExecutableHttpInput] [logstashrop4] failed to execute [http] input for watch [e45b8f3e-e2a
d-4b46-a394-be9611937823], reason [PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to fin
d valid certification path to requested target]
[2021-09-12T11:16:48,633][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [logstashrop4] http client did not trust this server's
certificate, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/10.198.134.174:57148}
[2021-09-12T11:21:48,805][WARN ][o.e.c.s.DiagnosticTrustManager] [logstashrop4] failed to establish trust with server at [10.198.134.17
4]; the server provided a certificate with subject name [CN=elasticsearch] and fingerprint [d80ef9807f2bfc042ab44214a223995145c9e334];
the certificate has subject alternative names [IP:10.198.134.173,IP:10.198.134.174]; the certificate is issued by [CN=Elastic Certifica
te Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [5a4d3ee4d6
6a274ad5c61b7536ce5cd53fd433aa]) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is not trusted in
 this ssl context ([(shared)])
sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unabl
e to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:385) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:290) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:321) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:221) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:129) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:90) [elasticsearch-ssl-co
nfig-7.9.2.jar:7.9.2]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:626) [?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:461) [?:?]
        at sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:361) [?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392) [?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) [?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:421) [?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:178) [?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:164) [?:?]
        at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152) [?:?]
        at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1063) [?:?]
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:402) [?:?]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:436) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:384) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:374) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:393) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:186) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) [httpclient-4.5.10.jar:4.5.10]
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:72) [httpclient-4.5.10.jar:4.5.10]
        at org.elasticsearch.xpack.watcher.common.http.HttpClient.lambda$execute$1(HttpClient.java:239) [x-pack-watcher-7.9.2.jar:7.9.2]
        at java.security.AccessController.doPrivileged(Native Method) [?:?]
        at org.elasticsearch.xpack.core.common.socket.SocketAccess.doPrivileged(SocketAccess.java:32) [x-pack-core-7.9.2.jar:7.9.2]
        at org.elasticsearch.xpack.watcher.common.http.HttpClient.execute(HttpCli
2

My guess is that you configured xpack.ssl.certificate_authorities or xpack.ssl.truststore.path to include this certificate.
When you do that you stop trusting the default JRE ca certs, and only trust the certs that you explicitly listed.

If that's the case, then your options are:

  • Add the corporate CA into your xpack.ssl.certificate_authorities / xpack.ssl.truststore.path
  • Change your ssl config, so that the monitoring cert is configured directly within the monitoring exporter and leave xpack.ssl.* unset.

cc @Larry_Gregory / @jportner for more inputs

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.