Does slack action require that we add certs to keystore?

Souciance_Eqdam_Rash · January 4, 2018, 10:15am

Hello,

I am trying to add a slack action to my watcher (using ES/Kibana 6.1.1). I am getting the error:
[2018-01-04T11:11:02,859][ERROR][o.e.x.n.s.SlackService ] [SELULT4729] failed to execute slack api http request
javax.net.ssl.SSLException: Unrecognized SSL message, plaintext connection?
at sun.security.ssl.InputRecord.handleUnknownRecord(InputRecord.java:710) ~[?:?]
at sun.security.ssl.InputRecord.read(InputRecord.java:527) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) ~[?:?]
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) ~[?:?]
at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[?:?]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:388) ~[?:?]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[?:?]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[?:?]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[?:?]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[?:?]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[?:?]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.lambda$execute$0(HttpClient.java:184) ~[?:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
at org.elasticsearch.xpack.common.socket.SocketAccess.doPrivileged(SocketAccess.java:44) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:184) ~[?:?]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:122) ~[?:?]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:76) ~[?:?]
at org.elasticsearch.xpack.watcher.actions.slack.ExecutableSlackAction.execute(ExecutableSlackAction.java:62) ~[?:?]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:155) ~[?:?]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:485) ~[?:?]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:315) ~[?:?]
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.lambda$executeWatch$1(TransportExecuteWatchAction.java:154) ~[?:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_152]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_152]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.1.jar:6.1.1]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]

Does that mean we need to add the slack certs to the elastic keystore? Or am I missing some config in my elasticsearch.yml that should enable this without adding the certs?

Thanks
Souciance

Souciance_Eqdam_Rash · January 4, 2018, 10:43am

From what I can see, elasticsearch is trying to call the slack url with http protocol rather than https. Is there a config parameter that needs to be added so that elastic uses https for https urls?

ikakavas · January 4, 2018, 2:57pm

No, this is not what's happening. Watcher will try to make a request to the url you have set in

xpack.notification.slack:
  account:
    <acount_name>:
      url:

Please share relevant parts of your elasticsearch.yml file so that we can help you further.

Souciance_Eqdam_Rash · January 4, 2018, 3:37pm

This is my elastic config:

xpack.notification.slack:
  account:
    monitoring:
      url: https://hooks.slack.com/services/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
      message_defaults:
        from: x-pack
        to: notifications
        attachment:
          fallback: "X-Pack Notification"
          color: "#36a64f"
          title: "X-Pack Notification"
          title_link: "https://www.elastic.co/guide/en/x-pack/current/index.html"
          text: "One of your watches generated this notification."

As you can see https is included in the url. However doesn't the error indicate than an http call is being made?

ikakavas · January 4, 2018, 3:54pm

Yes, you are right. I got mixed up with a similar thread that I was replying to today, sorry for that. Could it be that there is an http proxy in front of Elasticsearch ?

Souciance_Eqdam_Rash · January 4, 2018, 4:09pm

There is a proxy fronting elasticsearch (outside of my control) and I tried setting

xpack.http.proxy.host: <proxyhost>
xpack.http.proxy.port: 8080

which is when I got the errors. Are there parameters for https as well that you need to set?

I haven't found anything similar to:

xpack.https.proxy.host:
xpack.https.proxy.port: port

TimV · January 5, 2018, 2:06am

It actually indicates that a HTTPS call is being made, but the server it was connecting too didn't respond using TLS/SSL.
So Watcher is doing the right this as far as the https URL is concerned, but the proxy is causing an issue because it's pointing to an http proxy, not a https one.

You can include a specific proxy configuration within the slack action itself, and set it to use the SSL port for your proxy server.

Slack Action | X-Pack for the Elastic Stack [6.1] | Elastic

Souciance_Eqdam_Rash · January 5, 2018, 9:39am

Well after playing around with this a bit more I finally got beyond the previous error but now I get this instead.

failed to execute slack api http request
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1959) ~[?:?]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302) ~[?:?]
	at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296) ~[?:?]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1514) ~[?:?]
	at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216) ~[?:?]
	at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1026) ~[?:?]
	at sun.security.ssl.Handshaker.process_record(Handshaker.java:961) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1072) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:?]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:394) ~[?:?]
	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:353) ~[?:?]
	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:141) ~[?:?]
	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:353) ~[?:?]
	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:388) ~[?:?]
	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:236) ~[?:?]
	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184) ~[?:?]
	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88) ~[?:?]
	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[?:?]
	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[?:?]
	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82) ~[?:?]
	at org.elasticsearch.xpack.common.http.HttpClient.lambda$execute$0(HttpClient.java:184) ~[?:?]
	at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_152]
	at org.elasticsearch.xpack.common.socket.SocketAccess.doPrivileged(SocketAccess.java:44) ~[?:?]
	at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:184) ~[?:?]
	at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:122) ~[?:?]
	at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:76) ~[?:?]
	at org.elasticsearch.xpack.watcher.actions.slack.ExecutableSlackAction.execute(ExecutableSlackAction.java:62) ~[?:?]
	at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:155) ~[?:?]
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:485) ~[?:?]
	at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:315) ~[?:?]
	at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.lambda$executeWatch$1(TransportExecuteWatchAction.java:154) ~[?:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_152]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_152]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.1.jar:6.1.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_152]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_152]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_152]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[?:?]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:568) ~[?:?]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:?]
	... 35 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) ~[?:?]
	at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126) ~[?:?]
	at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280) ~[?:1.8.0_152]
	at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392) ~[?:?]
	at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302) ~[?:?]
	at sun.security.validator.Validator.validate(Validator.java:260) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:324) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:229) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:124) ~[?:?]
	at org.elasticsearch.xpack.ssl.SSLService$ReloadableTrustManager.checkServerTrusted(SSLService.java:568) ~[?:?]
	at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496) ~[?:?]
	... 35 more

For me, this error says to say that it is trying to find the certificates for the slack host but cannot find it. Does that mean the proxy is causing this issue or do we need to add the certs to the keystore?

TimV · January 8, 2018, 4:04am

It's not possible for us to tell from just that error message.

If your proxy is actually doing SSL interception, then it will be dynamically issuing new certificates for every site, and signing them using its own internal CA. If that's the case then you will need to configure X-Pack/Watcher to trust that CA.

However, that might not be the cause. It could also be that you've configured X-Pack to only trust a small set of CAs and the CA for Slack isn't in that set.

spinscale · January 8, 2018, 8:12am

One minor tidbit I saw in the wild: I have seen openjdk distribution packages shipped with too old CAs (I think it was ubuntu), so that recent certs are not included, resulting in the error above. Checking the age of your openjdk package or trying out oracle JDK might be something to test as well.

--Alex

Souciance_Eqdam_Rash · January 8, 2018, 10:11am

Hi, we are running Oracle jdk 1.8_0152 and not openjdk.

Souciance_Eqdam_Rash · January 8, 2018, 10:12am

Where can you check which set of CAs are trusted by X-Pack?

ikakavas · January 8, 2018, 10:04pm

If you have set any of the

xpack.ssl.certificate_authorities
xpack.security.http.ssl.certificate_authorities
xpack.ssl.truststore.path
xpack.security.http.ssl.truststore.path

settings in your elasticsearch.yml then CAs that are trusted are the ones contained in the file or the truststore that the setting points to. If not, the system/JVM truststore is used.

system · February 5, 2018, 10:04pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.