SSLHandshakeException while using webhook action in watcher


(Biswars) #1

Version: 5.4.2
I do have the CA cert used in the target host in the webhook action url and it is tested using curl --cacert option and works well.

Followed the followings
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/notification-settings.html#ssl-notification-settings

However, I still get "SSLHandshakeException[Received fatal alert: handshake_failure]" in the response

As per the logs (we can see below) the error seems to be in the permission of the trust store file:
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")

But the permission of the file /etc/pki/java/cacerts is "lrwxrwxrwx." (using ls -lht command)

Any suggestion here would really help

The logs seen is as follows:
[2017-07-28T11:49:54,321][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts, -Djavax.net.ssl.truststore.password=changeit, -Des.path.home=/usr/share/elasticsearch]
[2017-07-28T11:50:00,948][ERROR][o.e.b.Bootstrap ] Exception
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:430) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:383) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:139) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.node.Node.(Node.java:310) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.node.Node.(Node.java:242) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:232) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:350) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.2.jar:5.4.2]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_131]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:419) ~[elasticsearch-5.4.2.jar:5.4.2]
... 14 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:57) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:378) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:401) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.(SSLService.java:79) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.(XPackPlugin.java:205) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_131]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:419) ~[elasticsearch-5.4.2.jar:5.4.2]
... 14 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_131]
at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_131]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_131]
at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_131]
at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:245) ~[?:?]
at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:136) ~[?:?]
at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:148) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:212) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_131]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_131]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_131]
at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_131]
at org.elasticsearch.xpack.ssl.CertUtils.trustManager(CertUtils.java:164) ~[?:?]
at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:55) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:378) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:401) ~[?:?]


(Alexander Reelsen) #2

If you are using ES 5.x, then watcher.http.ssl.keystore is the wrong setting, as those have been moved over to the xpack namespace, so it should be xpack.http.ssl.keystore

However the exception stats something else in addition.

Caused by: java.security.AccessControlException: access denied (“java.io.FilePermission” “/etc/pki/java/cacerts” “read”)

You need to put the certs into the elasticsearch configuration directory, you cannt put them anywhere in the filesystems.

--Alex


(Biswars) #3

Thanks @spinscale . The file permission error is resolved now. Just a query out of my curiosity as the elasticsearch is running as the user "elasticsearch" and the certs file is present in a path that has full access, how was it still not accessible !!

I am yet to get over the "SSLHandshakeException[Received fatal alert: handshake_failure".
The relevant section in elasticsearch.yml looks like below

xpack.http.ssl.certificate_authorities: /etc/elasticsearch/cacert.pem (the CA cert)
xpack.ssl.client_authentication: none (Don't want to have the client cert presentation during the handshake)

Anything missing here ?


(Alexander Reelsen) #4

please supply exceptions (also check the logs of the master node on startup), everything is else just guessing


(Biswars) #5

The log shows the same exception "SSLHandshakeException" and I believe it is due to the CA cert is not trusted. I used the PEM encoded cert same as used in the target host (to which the web action is connecting to) setting it to the config param xpack.http.ssl.certificate_authorities which did not work either.

For reference, the following is the log in the master:
[2017-07-28T18:22:51,797][ERROR][o.e.x.w.a.w.ExecutableWebhookAction] [A0cH-TU] failed to execute action [inlined/t_webhook]
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) ~[?:?]
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.doExecute(HttpClient.java:171) ~[x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:92) ~[x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.actions.webhook.ExecutableWebhookAction.execute(ExecutableWebhookAction.java:59) ~[x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:160) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:412) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:143) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:65) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.2.jar:5.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]


(Alexander Reelsen) #6

Can you also share your elasticsearch.yml configuration, at least all the xpack. settings? Thanks!


(Biswars) #7

Here it is:
xpack.http.ssl.certificate_authorities: /etc/elasticsearch/cacert.pem (the CA cert)
xpack.ssl.client_authentication: none (Don’t want to have the client cert presentation during the handshake)


(Alexander Reelsen) #8

Hey,

I dug through the code and this is not supported currently. We need to add this though.

--Alex


(Biswars) #9

Thanks @spinscale for this confirmation.
So by any other means we can get over this by importing the cert and using -Djavax.net.ssl.trustStore option during the jvm startup ?


(Alexander Reelsen) #10

yes. you need to set up your own truststore and use the xpack.http.ssl settings to point to it. If that one is not configured, the standard ssl truststore will be used.


(Biswars) #11

Just to re-confirm, can I use xpack.http.ssl.keystore.path to point to my own truststore ?
Secondly, by standard truststore do you mean using the -Djavax.net.ssl.trustStore option during the jvm startup ?


(Alexander Reelsen) #12

Exactly, you can point to your own truststore. Sorry for not being clear, with standard truststore I mean the one, you configure under the xpack.ssl.truststore configuration namespace, that works also for security and the realms.


(Biswars) #13

I have tried using the configurations as mentioned above by setting xpack.ssl.truststore.path to the .jks file and xpack.ssl.truststore.password to the password of the truststore.

However, I still get exactly the same error as earlier. Anything missing still ?

Note: I have also tried using -Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStorePassword and the error received is still the same


(Biswars) #14

Have tried the same trustostore with a custom java client (using SSLSocketFactory) as well and it works fine. Is it that the truststore is completely ignored by elasticsearch even using -Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStorePassword?


(system) #15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.