Elasticsearch failed to execute slack api http request

miked · February 1, 2017, 4:07am

Hi there,

I'm using ES and Kibana 5.1.2 (with x-pack) and have configured a Watcher to send an alert to Slack. The Watcher and query is working but the following error is generated in ES when trying to send the message to Slack:

[2017-02-01T16:55:47,613][ERROR][o.e.x.n.s.SlackService ] [IODOv55] failed to execute slack api http request
javax.net.ssl.SSLException: java.security.ProviderException: java.security.KeyException
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) ~[?:?]
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1906) ~[?:?]
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1889) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1410) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.doExecute(HttpClient.java:160) ~[x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:90) ~[x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:123) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:78) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.actions.slack.ExecutableSlackAction.execute(ExecutableSlackAction.java:65) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:158) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:415) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:496) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.2.jar:5.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: java.security.ProviderException: java.security.KeyException
at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:147) ~[?:?]
at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:703) ~[?:1.8.0_101]
at sun.security.ssl.ECDHCrypt.(ECDHCrypt.java:77) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:721) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:281) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:?]
at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[?:?]
... 19 more
Caused by: java.security.KeyException
at sun.security.ec.ECKeyPairGenerator.generateECKeyPair(Native Method) ~[?:?]
at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:128) ~[?:?]
at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:703) ~[?:1.8.0_101]
at sun.security.ssl.ECDHCrypt.(ECDHCrypt.java:77) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:721) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:281) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:?]
at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[?:?]

This is my ES yml file, and as I thought it may be some issue with the security I just disabled as many of those settings as I thought appropriate, as you can see by the lines appended to the end:

xpack.security.enabled: false
xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: true

xpack.notification.slack:
account:
monitoring:
url: Unlock your productivity potential with Slack Platform | Slack
message_defaults:
from: x-pack

xpack.ssl.client_authentication: none
xpack.security.http.ssl.enabled: false
xpack.security.http.ssl.client_authentication: none
xpack.security.transport.ssl.enabled: false
xpack.security.transport.ssl.client_authentication: none

However, this is still not working, so appreciate any insights offered in helping me figure this out.
Thanks so much!

spinscale · February 1, 2017, 8:18am

Hey,

which java version are you using? How did you install that one (package, repo, etc)? And lastly what Linux distribution (plus version if possible) is this running on?

--Alex

miked · February 1, 2017, 9:25pm

Hi Alex,

Thanks for your response, and here are the details as requested:

[oracle@dbvrep01 elasticsearch-5.1.2]$ cat /etc/*-release
Oracle Linux Server release 6.3
Red Hat Enterprise Linux Server release 6.3 (Santiago)
Oracle Linux Server release 6.3
[oracle@dbvrep01 elasticsearch-5.1.2]$ uname -a
Linux dbvrep01 2.6.32-279.14.1.el6.x86_64 #1 SMP Tue Nov 6 11:18:01 PST 2012 x86_64 x86_64 x86_64 GNU/Linux

Java:

[oracle@dbvrep01 elasticsearch-5.1.2]$ java -version
openjdk version "1.8.0_101"
OpenJDK Runtime Environment (build 1.8.0_101-b13)
OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)

Did the offline install of x-pack with the zip file, as outlined in the following:

Thanks for your help with this.
Cheers!

spinscale · February 2, 2017, 8:16am

Hey,

what is returned when you run curl -v https://hooks.slack.com ; echo $?

I do not know much about oracle, but ubuntu has a ca-certificates-java package, that includes a bunch of certs that are needed to connect to certain hosts. Is Oracle Linux having something similar?

--Alex

miked · February 5, 2017, 12:26pm

Hi Alex,

To be honest I'm not sure about those certificates you mention on OEL...it's something I can look into, but if I run the command you supplied it seems to execute correctly:

[oracle@dbvrep01 ~]$ curl -v https://hooks.slack.com ; echo $?
About to connect() to hooks.slack.com port 443 (#0)

Trying 52.84.130.51... connected

Connected to hooks.slack.com (52.84.130.51) port 443 (#0)

Initializing NSS with certpath: sql:/etc/pki/nssdb

CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none

SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA

Server certificate:
subject: CN=*.slack.com,O="Slack Technologies, Inc.",L=San Francisco,ST=California,C=US
start date: Feb 01 00:00:00 2017 GMT
expire date: Feb 01 23:59:59 2019 GMT
common name: *.slack.com
issuer: CN=GeoTrust SSL CA - G3,O=GeoTrust Inc.,C=US
GET / HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Host: hooks.slack.com
Accept: /

< HTTP/1.1 302 Found
< Content-Type: text/html
< Content-Length: 0
< Connection: keep-alive
< Content-Security-Policy: referrer no-referrer;
< Date: Sun, 05 Feb 2017 12:21:53 GMT
< location: https://api.slack.com/
< Server: Apache
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Vary: Accept-Encoding
< X-Frame-Options: SAMEORIGIN
< X-Slack-Backend: h
< X-Cache: Miss from cloudfront
< Via: 1.1 f7526e0e5d23dbedce7f37483ef58a35.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: Z5oO3owU4wmDowtHkqzvRbx-YAlosgGx9mDOEE56aQffnb10Bedt-Q==
<

Connection #0 to host hooks.slack.com left intact

Closing connection #0
0

How does this look to you? Anything wrong/strange that you can see?

Thanks again for your assistance with this!

spinscale · February 6, 2017, 4:37pm

Hey,

I have the feeling that you are missing certificate authority certs in your default system keystore/truststore. Can you test with an oracle java version and report back?

--Alex

miked · February 16, 2017, 4:35am

Thanks for the suggestion Alex. Running with the Oracle JDK seems to have remedied the issue

Just for reference here's what the curl command returned once the Oracle JDK had been installed (and the OpenJDK removed):

[oracle@dbvrep01 alternatives]$ curl -v https://hooks.slack.com ; echo $? *
About to connect() to hooks.slack.com port 443 (#0)

Trying 52.2.63.185... connected

Connected to hooks.slack.com (52.2.63.185) port 443 (#0)

Initializing NSS with certpath: sql:/etc/pki/nssdb

CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none

SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA

Server certificate:
subject: CN=*.slack.com,O="Slack Technologies, Inc.",L=San Francisco,ST=California,C=US
start date: Feb 01 00:00:00 2017 GMT
expire date: Feb 01 23:59:59 2019 GMT
common name: *.slack.com
issuer: CN=GeoTrust SSL CA - G3,O=GeoTrust Inc.,C=US
GET / HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Host: hooks.slack.com
Accept: /

< HTTP/1.1 302 Found
< Content-Security-Policy: referrer no-referrer;
< Content-Type: text/html
< Date: Thu, 16 Feb 2017 04:00:08 GMT
< location: https://api.slack.com/
< Server: Apache
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Vary: Accept-Encoding
< X-Frame-Options: SAMEORIGIN
< X-Slack-Backend: h
< Content-Length: 0
< Connection: keep-alive
<

Connection #0 to host hooks.slack.com left intact

Closing connection #0
0 appletviewer appletviewer.1 cassandra ControlPanel extcheck extcheck.1 idlj idlj.1 jar jar.1 jarsigner jarsigner.1 java java.1 javac javac.1 javadoc javadoc.1 javafxpackager javafxpackager.1 javah javah.1 javap javap.1 javapackager javapackager.1 java-rmi.cgi javaws javaws.1 jcmd jcmd.1 jconsole jconsole.1 jcontrol jdb jdb.1 jdeps jdeps.1 jhat jhat.1 jinfo jinfo.1 jjs jjs.1 jmap jmap.1 jmc jmc.1 jmc.ini jps jps.1 jrunscript jrunscript.1 jsadebugd jsadebugd.1 jstack jstack.1 jstat jstat.1 jstatd jstatd.1 jvisualvm jvisualvm.1 keytool keytool.1 mta mta-aliasesman mta-mailq mta-mailqman mta-newaliases mta-newaliasesman mta-pam mta-rmail mta-sendmail mta-sendmailman native2ascii native2ascii.1 orbd orbd.1 pack200 pack200.1 policytool policytool.1 rmic rmic.1 rmid rmid.1 rmiregistry rmiregistry.1 schemagen schemagen.1 serialver serialver.1 servertool servertool.1 tnameserv tnameserv.1 unpack200 unpack200.1 wsgen wsgen.1 wsimport wsimport.1 xjc xjc.1

Thanks again for your helping in sorting this out - most appreciated.

prodrg · February 17, 2017, 3:35pm

Hello,

The same issue is happening to me. I have the same configuration as miked and I am using OpenJDK, but when I execute curl -v https://hooks.slack.com ; echo $?, it shows the certificate as on last answer.

Is it mandatory to use Oracle JDK, or is there something I need to check?

Regards

prodrg · February 17, 2017, 3:43pm

It seems that I also have to comment the SSL keys and certificates on elasticesarch.yml file. Issue solved for me.

spinscale · February 20, 2017, 1:12pm

@miked so apparently every openJDK packager (like the distributions) is free to choose what to put into the distribution. So even same openjdk versions from different distributions can contain different cacerts resulting in error messages from above.

I will add some paragraphs to the x-pack docs regarding that.

miked · February 21, 2017, 4:42am

Hi Alex,

Thanks for this - it is good to know.
Would be brilliant if you can add something about this in the docs.

Cheers!

system · March 21, 2017, 4:42am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.