Does slack action require that we add certs to keystore?

Hello,

I am trying to add a slack action to my watcher (using ES/Kibana 6.1.1). I am getting the error:
[2018-01-04T11:11:02,859][ERROR][o.e.x.n.s.SlackService ] [SELULT4729] failed to execute slack api http request
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) ~[?:?]
at sun.security.ssl.InputRecord.read(InputRecord.java:527) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) ~[?:?]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[?:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:388) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[?:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[?:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[?:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[?:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.lambda$execute$0(HttpClient.java:184) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
at org.elasticsearch.xpack.common.socket.SocketAccess.doPrivileged(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:184) ~[?:?]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:122) ~[?:?]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:76) ~[?:?]
at org.elasticsearch.xpack.watcher.actions.slack.ExecutableSlackAction.execute(ExecutableSlackAction.java:62) ~[?:?]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:155) ~[?:?]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:485) ~[?:?]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:315) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.lambda$executeWatch$1(TransportExecuteWatchAction.java:154) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_152]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.1.jar:6.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Does that mean we need to add the slack certs to the elastic keystore? Or am I missing some config in my elasticsearch.yml that should enable this without adding the certs?

Thanks
Souciance

From what I can see, elasticsearch is trying to call the slack url with http protocol rather than https. Is there a config parameter that needs to be added so that elastic uses https for https urls?

No, this is not what's happening. Watcher will try to make a request to the url you have set in

xpack.notification.slack:
  account:
    <acount_name>:
      url:

Please share relevant parts of your elasticsearch.yml file so that we can help you further.

This is my elastic config:

xpack.notification.slack:
  account:
    monitoring:
      url: https://hooks.slack.com/services/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
      message_defaults:
        from: x-pack
        to: notifications
        attachment:
          fallback: "X-Pack Notification"
          color: "#36a64f"
          title: "X-Pack Notification"
          title_link: "https://www.elastic.co/guide/en/x-pack/current/index.html"
          text: "One of your watches generated this notification."   

As you can see https is included in the url. However doesn't the error indicate than an http call is being made?

Yes, you are right. I got mixed up with a similar thread that I was replying to today, sorry for that. Could it be that there is an http proxy in front of Elasticsearch ?

There is a proxy fronting elasticsearch (outside of my control) and I tried setting

xpack.http.proxy.host: <proxyhost>
xpack.http.proxy.port: 8080

which is when I got the errors. Are there parameters for https as well that you need to set?

I haven't found anything similar to:

xpack.https.proxy.host:
xpack.https.proxy.port: port

It actually indicates that a HTTPS call is being made, but the server it was connecting too didn't respond using TLS/SSL.
So Watcher is doing the right this as far as the https URL is concerned, but the proxy is causing an issue because it's pointing to an http proxy, not a https one.

You can include a specific proxy configuration within the slack action itself, and set it to use the SSL port for your proxy server.

Well after playing around with this a bit more I finally got beyond the previous error but now I get this instead.

failed to execute slack api http request
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) ~[?:?]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[?:?]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:?]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) ~[?:?]
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:?]
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:?]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) ~[?:?]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) ~[?:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[?:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[?:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:388) ~[?:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[?:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[?:?]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[?:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[?:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[?:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[?:?]
	at org.elasticsearch.xpack.common.http.HttpClient.lambda$execute$0(HttpClient.java:184) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at org.elasticsearch.xpack.common.socket.SocketAccess.doPrivileged(SocketAccess.java:44) ~[?:?]
	at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:184) ~[?:?]
	at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:122) ~[?:?]
	at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:76) ~[?:?]
	at org.elasticsearch.xpack.watcher.actions.slack.ExecutableSlackAction.execute(ExecutableSlackAction.java:62) ~[?:?]
	at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:155) ~[?:?]
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:485) ~[?:?]
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:315) ~[?:?]
	at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.lambda$executeWatch$1(TransportExecuteWatchAction.java:154) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_152]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_152]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.1.jar:6.1.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[?:?]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:568) ~[?:?]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:?]
	... 35 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_152]
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[?:?]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:568) ~[?:?]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:?]
	... 35 more

For me, this error says to say that it is trying to find the certificates for the slack host but cannot find it. Does that mean the proxy is causing this issue or do we need to add the certs to the keystore?

It's not possible for us to tell from just that error message.

If your proxy is actually doing SSL interception, then it will be dynamically issuing new certificates for every site, and signing them using its own internal CA. If that's the case then you will need to configure X-Pack/Watcher to trust that CA.

However, that might not be the cause. It could also be that you've configured X-Pack to only trust a small set of CAs and the CA for Slack isn't in that set.

One minor tidbit I saw in the wild: I have seen openjdk distribution packages shipped with too old CAs (I think it was ubuntu), so that recent certs are not included, resulting in the error above. Checking the age of your openjdk package or trying out oracle JDK might be something to test as well.

--Alex

Hi, we are running Oracle jdk 1.8_0152 and not openjdk.

Where can you check which set of CAs are trusted by X-Pack?

If you have set any of the

xpack.ssl.certificate_authorities
xpack.security.http.ssl.certificate_authorities
xpack.ssl.truststore.path
xpack.security.http.ssl.truststore.path

settings in your elasticsearch.yml then CAs that are trusted are the ones contained in the file or the truststore that the setting points to. If not, the system/JVM truststore is used.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.