Elasticsearch failed to execute slack api http request

Hi there,

I'm using ES and Kibana 5.1.2 (with x-pack) and have configured a Watcher to send an alert to Slack. The Watcher and query is working but the following error is generated in ES when trying to send the message to Slack:

[2017-02-01T16:55:47,613][ERROR][o.e.x.n.s.SlackService ] [IODOv55] failed to execute slack api http request
javax.net.ssl.SSLException: java.security.ProviderException: java.security.KeyException
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949) ~[?:?]
at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1906) ~[?:?]
at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1889) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1410) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1283) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1258) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.doExecute(HttpClient.java:160) ~[x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:90) ~[x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:123) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.notification.slack.SlackAccount.send(SlackAccount.java:78) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.actions.slack.ExecutableSlackAction.execute(ExecutableSlackAction.java:65) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:158) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:415) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService$WatchExecutionTask.run(ExecutionService.java:496) [x-pack-5.1.2.jar:5.1.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:458) [elasticsearch-5.1.2.jar:5.1.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_101]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_101]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_101]
Caused by: java.security.ProviderException: java.security.KeyException
at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:147) ~[?:?]
at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:703) ~[?:1.8.0_101]
at sun.security.ssl.ECDHCrypt.(ECDHCrypt.java:77) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:721) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:281) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:?]
at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[?:?]
... 19 more
Caused by: java.security.KeyException
at sun.security.ec.ECKeyPairGenerator.generateECKeyPair(Native Method) ~[?:?]
at sun.security.ec.ECKeyPairGenerator.generateKeyPair(ECKeyPairGenerator.java:128) ~[?:?]
at java.security.KeyPairGenerator$Delegate.generateKeyPair(KeyPairGenerator.java:703) ~[?:1.8.0_101]
at sun.security.ssl.ECDHCrypt.(ECDHCrypt.java:77) ~[?:?]
at sun.security.ssl.ClientHandshaker.serverKeyExchange(ClientHandshaker.java:721) ~[?:?]
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:281) ~[?:?]
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979) ~[?:?]
at sun.security.ssl.Handshaker.process_record(Handshaker.java:914) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403) ~[?:?]

This is my ES yml file, and as I thought it may be some issue with the security I just disabled as many of those settings as I thought appropriate, as you can see by the lines appended to the end:

xpack.security.enabled: false
xpack.monitoring.enabled: false
xpack.graph.enabled: false
xpack.watcher.enabled: true

xpack.notification.slack:
account:
monitoring:
url: https://hooks.slack.com/services/
message_defaults:
from: x-pack

xpack.ssl.client_authentication: none
xpack.security.http.ssl.enabled: false
xpack.security.http.ssl.client_authentication: none
xpack.security.transport.ssl.enabled: false
xpack.security.transport.ssl.client_authentication: none

However, this is still not working, so appreciate any insights offered in helping me figure this out.
Thanks so much!

Hey,

which java version are you using? How did you install that one (package, repo, etc)? And lastly what Linux distribution (plus version if possible) is this running on?

--Alex

Hi Alex,

Thanks for your response, and here are the details as requested:

[oracle@dbvrep01 elasticsearch-5.1.2]$ cat /etc/*-release
Oracle Linux Server release 6.3
Red Hat Enterprise Linux Server release 6.3 (Santiago)
Oracle Linux Server release 6.3
[oracle@dbvrep01 elasticsearch-5.1.2]$ uname -a
Linux dbvrep01 2.6.32-279.14.1.el6.x86_64 #1 SMP Tue Nov 6 11:18:01 PST 2012 x86_64 x86_64 x86_64 GNU/Linux

Java:

[oracle@dbvrep01 elasticsearch-5.1.2]$ java -version
openjdk version "1.8.0_101"
OpenJDK Runtime Environment (build 1.8.0_101-b13)
OpenJDK 64-Bit Server VM (build 25.101-b13, mixed mode)

Did the offline install of x-pack with the zip file, as outlined in the following:
https://www.elastic.co/guide/en/x-pack/current/installing-xpack.html#xpack-package-installation

Thanks for your help with this.
Cheers!

Hey,

what is returned when you run curl -v https://hooks.slack.com ; echo $?

I do not know much about oracle, but ubuntu has a ca-certificates-java package, that includes a bunch of certs that are needed to connect to certain hosts. Is Oracle Linux having something similar?

--Alex

Hi Alex,

To be honest I'm not sure about those certificates you mention on OEL...it's something I can look into, but if I run the command you supplied it seems to execute correctly:

[oracle@dbvrep01 ~]$ curl -v https://hooks.slack.com ; echo $?

  • About to connect() to hooks.slack.com port 443 (#0)
  • Trying 52.84.130.51... connected
  • Connected to hooks.slack.com (52.84.130.51) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_RSA_WITH_AES_256_CBC_SHA
  • Server certificate:
  • subject: CN=*.slack.com,O="Slack Technologies, Inc.",L=San Francisco,ST=California,C=US
    
  • start date: Feb 01 00:00:00 2017 GMT
    
  • expire date: Feb 01 23:59:59 2019 GMT
    
  • common name: *.slack.com
    
  • issuer: CN=GeoTrust SSL CA - G3,O=GeoTrust Inc.,C=US
    

GET / HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Host: hooks.slack.com
Accept: /

< HTTP/1.1 302 Found
< Content-Type: text/html
< Content-Length: 0
< Connection: keep-alive
< Content-Security-Policy: referrer no-referrer;
< Date: Sun, 05 Feb 2017 12:21:53 GMT
< location: https://api.slack.com/
< Server: Apache
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Vary: Accept-Encoding
< X-Frame-Options: SAMEORIGIN
< X-Slack-Backend: h
< X-Cache: Miss from cloudfront
< Via: 1.1 f7526e0e5d23dbedce7f37483ef58a35.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: Z5oO3owU4wmDowtHkqzvRbx-YAlosgGx9mDOEE56aQffnb10Bedt-Q==
<

  • Connection #0 to host hooks.slack.com left intact
  • Closing connection #0
    0

How does this look to you? Anything wrong/strange that you can see?

Thanks again for your assistance with this!

Hey,

I have the feeling that you are missing certificate authority certs in your default system keystore/truststore. Can you test with an oracle java version and report back?

--Alex

Thanks for the suggestion Alex. Running with the Oracle JDK seems to have remedied the issue :slight_smile:

Just for reference here's what the curl command returned once the Oracle JDK had been installed (and the OpenJDK removed):

[oracle@dbvrep01 alternatives]$ curl -v https://hooks.slack.com ; echo $? *

  • About to connect() to hooks.slack.com port 443 (#0)
  • Trying 52.2.63.185... connected
  • Connected to hooks.slack.com (52.2.63.185) port 443 (#0)
  • Initializing NSS with certpath: sql:/etc/pki/nssdb
  • CAfile: /etc/pki/tls/certs/ca-bundle.crt
    CApath: none
  • SSL connection using TLS_RSA_WITH_AES_128_CBC_SHA
  • Server certificate:
  • subject: CN=*.slack.com,O="Slack Technologies, Inc.",L=San Francisco,ST=California,C=US
    
  • start date: Feb 01 00:00:00 2017 GMT
    
  • expire date: Feb 01 23:59:59 2019 GMT
    
  • common name: *.slack.com
    
  • issuer: CN=GeoTrust SSL CA - G3,O=GeoTrust Inc.,C=US
    

GET / HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-unknown-linux-gnu) libcurl/7.19.7 NSS/3.13.1.0 zlib/1.2.3 libidn/1.18 libssh2/1.2.2
Host: hooks.slack.com
Accept: /

< HTTP/1.1 302 Found
< Content-Security-Policy: referrer no-referrer;
< Content-Type: text/html
< Date: Thu, 16 Feb 2017 04:00:08 GMT
< location: https://api.slack.com/
< Server: Apache
< Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
< Vary: Accept-Encoding
< X-Frame-Options: SAMEORIGIN
< X-Slack-Backend: h
< Content-Length: 0
< Connection: keep-alive
<

  • Connection #0 to host hooks.slack.com left intact
  • Closing connection #0
    0 appletviewer appletviewer.1 cassandra ControlPanel extcheck extcheck.1 idlj idlj.1 jar jar.1 jarsigner jarsigner.1 java java.1 javac javac.1 javadoc javadoc.1 javafxpackager javafxpackager.1 javah javah.1 javap javap.1 javapackager javapackager.1 java-rmi.cgi javaws javaws.1 jcmd jcmd.1 jconsole jconsole.1 jcontrol jdb jdb.1 jdeps jdeps.1 jhat jhat.1 jinfo jinfo.1 jjs jjs.1 jmap jmap.1 jmc jmc.1 jmc.ini jps jps.1 jrunscript jrunscript.1 jsadebugd jsadebugd.1 jstack jstack.1 jstat jstat.1 jstatd jstatd.1 jvisualvm jvisualvm.1 keytool keytool.1 mta mta-aliasesman mta-mailq mta-mailqman mta-newaliases mta-newaliasesman mta-pam mta-rmail mta-sendmail mta-sendmailman native2ascii native2ascii.1 orbd orbd.1 pack200 pack200.1 policytool policytool.1 rmic rmic.1 rmid rmid.1 rmiregistry rmiregistry.1 schemagen schemagen.1 serialver serialver.1 servertool servertool.1 tnameserv tnameserv.1 unpack200 unpack200.1 wsgen wsgen.1 wsimport wsimport.1 xjc xjc.1

Thanks again for your helping in sorting this out - most appreciated.

Hello,

The same issue is happening to me. I have the same configuration as miked and I am using OpenJDK, but when I execute curl -v https://hooks.slack.com ; echo $?, it shows the certificate as on last answer.

Is it mandatory to use Oracle JDK, or is there something I need to check?

Regards

It seems that I also have to comment the SSL keys and certificates on elasticesarch.yml file. Issue solved for me.

@miked so apparently every openJDK packager (like the distributions) is free to choose what to put into the distribution. So even same openjdk versions from different distributions can contain different cacerts resulting in error messages from above.

I will add some paragraphs to the x-pack docs regarding that.

Hi Alex,

Thanks for this - it is good to know.
Would be brilliant if you can add something about this in the docs.

Cheers!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.