I am trying to add a slack action to my watcher (using ES/Kibana 6.1.1). I am getting the error:
[2018-01-04T11:11:02,859][ERROR][o.e.x.n.s.SlackService ] [SELULT4729] failed to execute slack api http request
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) ~[?:?]
at sun.security.ssl.InputRecord.read(InputRecord.java:527) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) ~[?:?]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[?:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:388) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[?:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[?:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[?:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[?:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.lambda$execute$0(HttpClient.java:184) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
at org.elasticsearch.xpack.common.socket.SocketAccess.doPrivileged(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:184) ~[?:?]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:122) ~[?:?]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:76) ~[?:?]
at org.elasticsearch.xpack.watcher.actions.slack.ExecutableSlackAction.execute(ExecutableSlackAction.java:62) ~[?:?]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:155) ~[?:?]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:485) ~[?:?]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:315) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.lambda$executeWatch$1(TransportExecuteWatchAction.java:154) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_152]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.1.jar:6.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Does that mean we need to add the slack certs to the elastic keystore? Or am I missing some config in my elasticsearch.yml that should enable this without adding the certs?
From what I can see, elasticsearch is trying to call the slack url with http protocol rather than https. Is there a config parameter that needs to be added so that elastic uses https for https urls?
Yes, you are right. I got mixed up with a similar thread that I was replying to today, sorry for that. Could it be that there is an http proxy in front of Elasticsearch ?
It actually indicates that a HTTPS call is being made, but the server it was connecting too didn't respond using TLS/SSL.
So Watcher is doing the right this as far as the https URL is concerned, but the proxy is causing an issue because it's pointing to an http proxy, not a https one.
You can include a specific proxy configuration within the slack action itself, and set it to use the SSL port for your proxy server.
Well after playing around with this a bit more I finally got beyond the previous error but now I get this instead.
failed to execute slack api http request
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[?:?]
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:?]
at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) ~[?:?]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[?:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:388) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[?:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[?:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[?:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[?:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.lambda$execute$0(HttpClient.java:184) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
at org.elasticsearch.xpack.common.socket.SocketAccess.doPrivileged(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:184) ~[?:?]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:122) ~[?:?]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:76) ~[?:?]
at org.elasticsearch.xpack.watcher.actions.slack.ExecutableSlackAction.execute(ExecutableSlackAction.java:62) ~[?:?]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:155) ~[?:?]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:485) ~[?:?]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:315) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.lambda$executeWatch$1(TransportExecuteWatchAction.java:154) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_152]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.1.jar:6.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:568) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:?]
... 35 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_152]
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:568) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:?]
... 35 more
For me, this error says to say that it is trying to find the certificates for the slack host but cannot find it. Does that mean the proxy is causing this issue or do we need to add the certs to the keystore?
It's not possible for us to tell from just that error message.
If your proxy is actually doing SSL interception, then it will be dynamically issuing new certificates for every site, and signing them using its own internal CA. If that's the case then you will need to configure X-Pack/Watcher to trust that CA.
However, that might not be the cause. It could also be that you've configured X-Pack to only trust a small set of CAs and the CA for Slack isn't in that set.
One minor tidbit I saw in the wild: I have seen openjdk distribution packages shipped with too old CAs (I think it was ubuntu), so that recent certs are not included, resulting in the error above. Checking the age of your openjdk package or trying out oracle JDK might be something to test as well.
settings in your elasticsearch.yml then CAs that are trusted are the ones contained in the file or the truststore that the setting points to. If not, the system/JVM truststore is used.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.