Watcher for report with attachment failed due to ssl handshake exception

vinayak · June 27, 2020, 3:07am

Unable to send reporting attachment by the help of watcher. My watch query : 

PUT _watcher/watch/sample_report_ssl
{
  "trigger" : {
    "schedule": {
      "interval": "1m"
    }
  },
  "actions" : {
    "email_admin" : { 
      "email": {
        "from": "*******@gmail.com",
        "to": "*********@gmail.com",
        "subject": "Error Report",
        "attachments" : {
          "error_report.pdf" : {
            "reporting" : {
              "url": "Copied Post URL from Share tab", 
              "retries":40, 
              "interval":"15s", 
              "auth":{ 
                "basic":{
                  "username":"abc",
                  "password":"******"
                }
              }
            }
          }
        }
      }
    }
  }
}

resulting exception:

 "actions": [
      {
        "id": "email_admin",
        "type": "email",
        "status": "failure",
        "error": {
          "root_cause": [
            {
              "type": "s_s_l_handshake_exception",
              "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
            }
          ],
          "type": "s_s_l_handshake_exception",
          "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
          "caused_by": {
            "type": "validator_exception",
            "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
            "caused_by": {
              "type": "sun_cert_path_builder_exception",
              "reason": "unable to find valid certification path to requested target"


My elasticsearch configuration looks like : 

xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/node-1.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/node-1.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/node-1.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/node-1.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
#########################################################
xpack.http.ssl.truststore.path: /etc/elasticsearch/certs/watcher-truststore.jks
xpack.http.ssl.truststore.password: ******
xpack.security.audit.enabled: true
xpack.watcher.enabled: true

Kibana Configuration is : 

xpack.reporting.enabled: true
#watcher setting for SSL/TLS based reporting
server.ssl.enabled: true
#elasticsearch.ssl.verificationMode: certificate
server.ssl.certificate: "/etc/kibana/config/certs/kibana-1.crt"
server.ssl.key: "/etc/kibana/config/certs/kibana-1.key"
elasticsearch.ssl.certificateAuthorities: ["/etc/kibana/config/certs/ca.crt"]
elasticsearch.ssl.verificationMode: certificate

I am following this article : https://www.elastic.co/guide/en/kibana/7.3/securing-reporting.html

Please suggest, thanking you in advance.

Julien · June 28, 2020, 10:02pm

As you use self-signed certificate in kibana, you need to add the CA certificate used to generate the kibana certificate for by watcher for https connection to kibana reporting in elasticsearch.yml
In this case this is setting called xpack.http.ssl.certificate_authorities which is currently missing or ensure that ca.crt for kibana certicate is included in /etc/elasticsearch/certs/watcher-truststore.jks

vinayak · June 29, 2020, 7:06am

Hi Julien, 
Thanks for your reply. For kibana, i have below mentioned self sign certificate, key and CA certificate available : 
(a) /etc/kibana/config/certs/kibana-1.crt
(b) /etc/kibana/config/certs/kibana-1.key
(c) /etc/kibana/config/certs/ca.crt**

Now as per your advice,if i am adding  , xpack.http.ssl.certificate_autorities, in my elasticsearch.yml file like this:

xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/node-1.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/node-1.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/node-1.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/node-1.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.http.ssl.certificate_authorities: [ "/etc/kibana/config/certs/ca.cert" ]
#########################################################
xpack.http.ssl.truststore.path: /etc/elasticsearch/certs/watcher-truststore.jks
xpack.http.ssl.truststore.password: *****

unable to start the elasticsearch, for generating watcher-truststore.jks i have used /etc/kibana/config/certs/kibana-1.crt certificate, it is giving below mentioned error:

[2020-06-29T01:31:59,137][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [node-1] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: ElasticsearchSecurityException[failed to load SSL configuration [xpack.http.ssl]]; nested: IllegalArgumentException[you cannot specify a truststore and ca files];
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:174) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:161) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125) ~[elasticsearch-cli-7.6.0.jar:7.6.0]
	at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-7.6.0.jar:7.6.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:126) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-7.6.0.jar:7.6.0]
Caused by: org.elasticsearch.ElasticsearchSecurityException: failed to load SSL configuration [xpack.http.ssl]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadConfiguration(SSLService.java:524) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:497) ~[?:?]
	at java.util.HashMap.forEach(HashMap.java:1338) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadSSLConfigurations(SSLService.java:497) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:142) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.<init>(SSLService.java:130) ~[?:?]
	at org.elasticsearch.xpack.core.XPackPlugin.createComponents(XPackPlugin.java:259) ~[?:?]
	at org.elasticsearch.node.Node.lambda$new$9(Node.java:456) ~[elasticsearch-7.6.0.jar:7.6.0]
	at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) ~[?:?]
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1621) ~[?:?]
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
	at org.elasticsearch.node.Node.<init>(Node.java:459) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.node.Node.<init>(Node.java:257) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:221) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:349) ~[elasticsearch-7.6.0.jar:7.6.0]
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:170) ~[elasticsearch-7.6.0.jar:7.6.0]
	... 6 more
Caused by: java.lang.IllegalArgumentException: you cannot specify a truststore and ca files
	at org.elasticsearch.xpack.core.ssl.SSLConfiguration.createCertChainTrustConfig(SSLConfiguration.java:181) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLConfiguration.createTrustConfig(SSLConfiguration.java:170) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLConfiguration.<init>(SSLConfiguration.java:50) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadConfiguration(SSLService.java:519) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.lambda$loadSSLConfigurations$4(SSLService.java:497) ~[?:?]
	at java.util.HashMap.forEach(HashMap.java:1338) ~[?:?]

so , as per error, i have changed the elasticsearch.yml configuration as:

xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/node-1.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/node-1.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/node-1.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/node-1.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.http.ssl.certificate_authorities: [ "/etc/kibana/config/certs/ca.cert" ]

Now it is giving below mentioned error:

Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager - access to read certificate_authorities file [/etc/kibana/config/certs/ca.cert] is blocked; SSL resources should be placed in the [/etc/elasticsearch] directory
	at org.elasticsearch.xpack.core.ssl.TrustConfig.blockedTrustConfigFile(TrustConfig.java:135) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.PEMTrustConfig.createTrustManager(PEMTrustConfig.java:61) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.createSslContext(SSLService.java:427) ~[?:?]
	at java.util.HashMap.computeIfAbsent(HashMap.java:1138) ~[?:?]
	at org.elasticsearch.xpack.core.ssl.SSLService.loadConfiguration(SSLService.java:521) ~[?:?]
	... 26 more
Caused by: java.security.AccessControlException: access denied ("java.io.FilePermission" "/etc/kibana/config/certs/ca.cert" "read")
	at java.security.AccessControlContext.checkPermission(AccessControlContext.java:472) ~[?:?]
	at java.security.AccessController.checkPermission(AccessController.java:1036) ~[?:?]
	at java.lang.SecurityManager.checkPermission(SecurityManager.java:408) ~[?:?]
	at java.lang.SecurityManager.checkRead(SecurityManager.java:747) ~[?:?]
	at sun.nio.fs.UnixChannelFactory.open(UnixChannelFactory.java:255) ~[?:?]
	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:143) ~[?:?]
	at sun.nio.fs.UnixChannelFactory.newFileChannel(UnixChannelFactory.java:156) ~[?:?]
	at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:217) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:374) ~[?:?]
	at java.nio.file.Files.newByteChannel(Files.java:425) ~[?:?]

by checking this error come to know that, /etc/kibana/config/certs/ca.crt having permission issue, and suggesting that we need to place the file at /etc/elasticsearch location, so i have placed the file ca.crt file at this location using below mentioned command:

ubuntu@ip-172-31-13-56:~$ cp /etc/kibana/config/certs/ca.crt /etc/elasticsearch/
ubuntu@ip-172-31-13-56:/etc/elasticsearch$ ll ca.crt
-rwxrwxr-x 1 ubuntu elasticsearch 1200 Jun 29 06:26 ca.crt*

Now again i have changed the configuration of elasticsearch.yml file as:

xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/certs/node-1.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/node-1.crt
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/node-1.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/node-1.crt
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.http.ssl.certificate_authorities: [ "/etc/elasticsearch/ca.crt" ]

And, now it is working. Thnx so much...........

vinayak · June 29, 2020, 7:08am

Please ignore spelling mistake in configuration, that was blunder........

system · July 27, 2020, 7:09am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
S_s_l_handshake_exception in watcher Elasticsearch elastic-stack-reporting , elastic-stack-alerting	1	601	November 21, 2022
Watcher ssl fail Kibana elastic-stack-security , elastic-stack-alerting	2	257	March 4, 2024
Kibana watcher error throwing SSL handshake even though CA is same for both Kibana & Elasticsearch Kibana elastic-stack-alerting	2	504	July 17, 2023
Handshake error attaching report Kibana	2	369	August 24, 2020
Watcher gives SSL handshake exception Elasticsearch elastic-stack-alerting	2	1114	December 25, 2018

Watcher for report with attachment failed due to ssl handshake exception

Related topics