Version: 5.4.2
I do have the CA cert used in the target host in the webhook action url and it is tested using curl --cacert option and works well.
Followed the followings
However, I still get "SSLHandshakeException[Received fatal alert: handshake_failure]" in the response
As per the logs (we can see below) the error seems to be in the permission of the trust store file:
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")
But the permission of the file /etc/pki/java/cacerts is "lrwxrwxrwx." (using ls -lht command)
Any suggestion here would really help
The logs seen is as follows:
[2017-07-28T11:49:54,321][INFO ][o.e.n.Node ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+DisableExplicitGC, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Djavax.net.ssl.trustStore=/etc/pki/java/cacerts, -Djavax.net.ssl.truststore.password=changeit, -Des.path.home=/usr/share/elasticsearch]
[2017-07-28T11:50:00,948][ERROR][o.e.b.Bootstrap ] Exception
org.elasticsearch.ElasticsearchException: Failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:430) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:383) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.plugins.PluginsService.(PluginsService.java:139) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.node.Node.(Node.java:310) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.node.Node.(Node.java:242) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Bootstrap$5.(Bootstrap.java:232) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:232) ~[elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:350) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:123) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:114) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:67) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:122) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.cli.Command.main(Command.java:88) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:91) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:84) [elasticsearch-5.4.2.jar:5.4.2]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_131]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:419) ~[elasticsearch-5.4.2.jar:5.4.2]
... 14 more
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize a TrustManagerFactory
at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:57) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:378) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:401) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.(SSLService.java:79) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.(XPackPlugin.java:205) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_131]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:419) ~[elasticsearch-5.4.2.jar:5.4.2]
... 14 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/pki/java/cacerts" "read")
at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:1.8.0_131]
at java.security.AccessController.checkPermission(AccessController.java:884) ~[?:1.8.0_131]
at java.lang.SecurityManager.checkPermission(SecurityManager.java:549) ~[?:1.8.0_131]
at java.lang.SecurityManager.checkRead(SecurityManager.java:888) ~[?:1.8.0_131]
at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:245) ~[?:?]
at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:136) ~[?:?]
at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:148) ~[?:?]
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:212) ~[?:?]
at java.nio.file.Files.newByteChannel(Files.java:361) ~[?:1.8.0_131]
at java.nio.file.Files.newByteChannel(Files.java:407) ~[?:1.8.0_131]
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384) ~[?:1.8.0_131]
at java.nio.file.Files.newInputStream(Files.java:152) ~[?:1.8.0_131]
at org.elasticsearch.xpack.ssl.CertUtils.trustManager(CertUtils.java:164) ~[?:?]
at org.elasticsearch.xpack.ssl.StoreTrustConfig.createTrustManager(StoreTrustConfig.java:55) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.createSslContext(SSLService.java:378) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:401) ~[?:?]
If you are using ES 5.x, then watcher.http.ssl.keystore is the wrong setting, as those have been moved over to the xpack namespace, so it should be xpack.http.ssl.keystore
However the exception stats something else in addition.
Thanks @spinscale . The file permission error is resolved now. Just a query out of my curiosity as the elasticsearch is running as the user "elasticsearch" and the certs file is present in a path that has full access, how was it still not accessible !!
I am yet to get over the "SSLHandshakeException[Received fatal alert: handshake_failure".
The relevant section in elasticsearch.yml looks like below
xpack.http.ssl.certificate_authorities: /etc/elasticsearch/cacert.pem (the CA cert)
xpack.ssl.client_authentication: none (Don't want to have the client cert presentation during the handshake)
The log shows the same exception "SSLHandshakeException" and I believe it is due to the CA cert is not trusted. I used the PEM encoded cert same as used in the target host (to which the web action is connecting to) setting it to the config param xpack.http.ssl.certificate_authorities which did not work either.
For reference, the following is the log in the master:
[2017-07-28T18:22:51,797][ERROR][o.e.x.w.a.w.ExecutableWebhookAction] [A0cH-TU] failed to execute action [inlined/t_webhook]
javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) ~[?:?]
at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2033) ~[?:?]
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135) ~[?:?]
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413) ~[?:?]
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397) ~[?:?]
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559) ~[?:?]
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316) ~[?:?]
at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291) ~[?:?]
at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250) ~[?:?]
at org.elasticsearch.xpack.common.http.HttpClient.doExecute(HttpClient.java:171) ~[x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.common.http.HttpClient.execute(HttpClient.java:92) ~[x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.actions.webhook.ExecutableWebhookAction.execute(ExecutableWebhookAction.java:59) ~[x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.actions.ActionWrapper.execute(ActionWrapper.java:160) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:412) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:143) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:65) [x-pack-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638) [elasticsearch-5.4.2.jar:5.4.2]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.4.2.jar:5.4.2]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_131]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_131]
Here it is:
xpack.http.ssl.certificate_authorities: /etc/elasticsearch/cacert.pem (the CA cert)
xpack.ssl.client_authentication: none (Don’t want to have the client cert presentation during the handshake)
Thanks @spinscale for this confirmation.
So by any other means we can get over this by importing the cert and using -Djavax.net.ssl.trustStore option during the jvm startup ?
yes. you need to set up your own truststore and use the xpack.http.ssl settings to point to it. If that one is not configured, the standard ssl truststore will be used.
Just to re-confirm, can I use xpack.http.ssl.keystore.path to point to my own truststore ?
Secondly, by standard truststore do you mean using the -Djavax.net.ssl.trustStore option during the jvm startup ?
Exactly, you can point to your own truststore. Sorry for not being clear, with standard truststore I mean the one, you configure under the xpack.ssl.truststore configuration namespace, that works also for security and the realms.
I have tried using the configurations as mentioned above by setting xpack.ssl.truststore.path to the .jks file and xpack.ssl.truststore.password to the password of the truststore.
However, I still get exactly the same error as earlier. Anything missing still ?
Note: I have also tried using -Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStorePassword and the error received is still the same
Have tried the same trustostore with a custom java client (using SSLSocketFactory) as well and it works fine. Is it that the truststore is completely ignored by elasticsearch even using -Djavax.net.ssl.trustStore and -Djavax.net.ssl.trustStorePassword?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.