Watcher for Reporting shows an SSL Exception in the output

Elastic Stack Kibana
1

Hello,

I am trying to create a watcher which generates daily/weekly reports of a dashboard and send that via emails. When the watcher is getting executed, it shows an SSL exception in the output (shown in the picture)

31%20PM
Screen Shot 2018-06-27 at 1.41.31 PM.png1071×417 66.8 KB

Below is the watcher I am using for this:

{
  "trigger": {
    "schedule": {
      "interval": "30s"
    }
  },
  "input": {
    "none": {}
  },
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "attachments": {
          "error.pdf": {
            "reporting": {
              "url": "https://xyz",
              "retries": 5,
              "interval": "15s"
            }
          }
        },
        "to": [
          "myemail@mydomain.com"
        ],
        "subject": "Weekly Monitoring Report"
      }
    }
  }
}

Please help me understand why this error is occurring here and how to solve this. Thanks in advance.

2

Hi,

The problem happens when watcher attempts to download the file from https://xyz as it cannot validate the TLS certificate that is presented for https://xyz

I am assuming the URL is pointing to your Kibana instance. What are your SSL settings in Kibana? Are you using a self signed certificate for TLS in Kibana ? ( Step 1 in this https://www.elastic.co/guide/en/kibana/6.3/configuring-tls.html#configuring-tls )

3

@ikakavas no, I am not using any self-signed certificate. Could that cause this problem?

4

No, not necessarily, I was just probing for what your setup looks like since that was not included in the original information.

The issue is as I describe above, and the generic solution is to make sure that watcher trusts your certificate or the certificate authority that has signed it . It is sadly not possible to give you a specific suggestion without knowing the rest of your configuration but note that Watcher ssl settings with regards to what certificates are trusted are described here https://www.elastic.co/guide/en/elasticsearch/reference/current/notification-settings.html#_watcher_tls_ssl_key_and_trusted_certificate_settings . If you haven't explicitly set that then it falls back to your node's SSL settings ( xpack.ssl.X) . If those haven't been set either, then it falls back to the truststore of your JVM ( the cacert file that ships with your java installation ).

If you can share some more details regarding

  1. What kind of a certificate you're using for kibana ( you said it's not self signed ,but is it signed by a public CA or an internal one ? ) I'd you don't know and you have access to openssl binary in your environment you can run
openssl s_client -connect xyz:443

and share the output, or at least your gained insights from running it.

  1. Your SSL configuration in elasticsearch.yml

we would be able to offer some more specific suggestions

1 Like
5

Thanks for responding.

  1. I am using a public CA certificate.
  2. My SSL configuration in elasticsearch.yml is below:
xpack.ssl.keystore.path: certs-path
xpack.ssl.truststore.path: certs-path
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate
6

@ikakavas I'm helping @swathi on this issue so might be able to help out some. The are using public CA for the SSL cert to the kibana UI. We are using self-signed certs for the node to node communications.

Here is our kibana reporting settings(i've removed the key but it's defined) :

xpack.reporting.enabled: true
xpack.reporting.encryptionKey:
xpack.reporting.kibanaServer.port: 443
xpack.reporting.kibanaServer.protocol: https
xpack.reporting.kibanaServer.hostname: metrics.xyz.com

7

This is the cause of your problem.
This tells X-Pack to only trust "certs-path" and not trust the JRE's defaults CAs.
It is almost always a mistake to configure xpack.ssl.truststore.path or xpack.ssl.certificate_authorities. (I accept that our docs used to recommend that you do this - they shouldn't have).

You should instead set your config to be:

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: certs-path
xpack.security.transport.ssl.truststore.path: certs-path
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs-path

You almost certainly don't want to have this setting:

xpack.security.http.ssl.verification_mode: certificate

The settings below are all related to trust, and for HTTP that means they affect how PKI (client certificates) work, but don't affect the way SSL works for the server itself. Unless you are trying to configure a particular setup for PKI, you probably don't want to configure them for HTTP (but you typically do want them for transport).

  • verification_mode
  • truststore.path
  • certificate_authorities
8

OK, I've made these change to the config, however I'm still receiving the same Error when I try to execute the watch.

9

xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.keystore.path: certs-path xpack.security.transport.ssl.truststore.path: certs-path xpack.security.transport.ssl.verification_mode: certificate xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: certs-path

10

Do you have any more details on the exception - it's possible that it's coming from your SMTP server rather than Kibana. Are you accessing the email server over SSL?

11

can you also please share the full output of the execute watch API or a full watch history entry? Thanks a lot!

12

This was my original thinking since the error was inside the email block of the script. However in other watcher's I am able to send simple emails, without attached reports. I've also verified that I'm not using SSL to the mail server.

13

Here is the output of a test execute. Thanks!

{
"watch_id": "_inlined_",
"node": "JBNgvNcpR_COY5YvJNncbQ",
"state": "executed",
"status": {
  "state": {
    "active": true,
    "timestamp": "2018-07-17T15:23:15.102Z"
  },
  "last_checked": "2018-07-17T15:23:15.103Z",
  "last_met_condition": "2018-07-17T15:23:15.103Z",
  "actions": {
    "email_admin": {
      "ack": {
        "timestamp": "2018-07-17T15:23:15.102Z",
        "state": "awaits_successful_execution"
      },
      "last_execution": {
        "timestamp": "2018-07-17T15:23:15.103Z",
        "successful": false,
        "reason": ""
      }
    }
},
"execution_state": "executed",
"version": -1
},
"trigger_event": {
"type": "manual",
"triggered_time": "2018-07-17T15:23:15.103Z",
"manual": {
  "schedule": {
    "scheduled_time": "2018-07-17T15:23:15.103Z"
  }
}
},
"input": {
"none": {}
 },
"condition": {
  "always": {}
},
"metadata": {
  "xpack": {
    "type": "json"
  }
},
"result": {
  "execution_time": "2018-07-17T15:23:15.103Z",
  "execution_duration": 7,
  "input": {
    "type": "none",
    "status": "success",
    "payload": {}
  },
  "condition": {
    "type": "always",
    "status": "success",
    "met": true
  },
  "actions": [
  {
    "id": "email_admin",
    "type": "email",
    "status": "failure",
    "error": {
      "root_cause": [
        {
          "type": "s_s_l_handshake_exception",
          "reason": "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
        }
      ],
      "type": "s_s_l_handshake_exception",
      "reason": "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
      "caused_by": {
        "type": "validator_exception",
        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
        "caused_by": {
          "type": "sun_cert_path_builder_exception",
          "reason": "unable to find valid certification path to requested target"
        }
      }
    }
  }
]
},
"messages": []
}
14

@jspeer as Tim already asked: Is it possible that you are using custom certificates with your mail server?

15

Here is my settings for the mail server:
xpack.notification.email.account:
smtp_account:
profile: standard
email_defaults:
from: svcs@xyz.com
smtp:
auth: false
starttls.enable: false
host: mailhost.xyz.com
port: 25

Also, I have emails being sent from other watches that are delivered properly. Wouldn't a misconfigured email certificate give the same error in that case?

16

Hey,

indeed it would. Are you using custom certificates for TLS communication in Kibana as well? I assume that accessing kibana is causing this.

Can you remove the reporting attachment from that watch to verify if this is the case?

--Alex

17

I've removed the report section, and the email sends without a problem. So there is something about this portion:

"attachments": { "error.pdf": { "reporting": { "url": "https://vccn7:5601/api/reporting/jobs/download/jj0j4zsp0zw290eb70b9g26e", "retries": 5, "interval": "15s" } } },

There are a few certs in play here. The first is for node to node elasticsearch communication, and it is all self-signed. Next is our private CA signed cert for https connections via nginx or kibana. The last is for the public https URL metrics.xyz.com and it is signed by a public CA (this terminates on a load balancer).

I've replaced the URL inside the watcher with all 3 possiblities for the URL:
kibana: https://vccn7:5601/api/reporting/jobs/download/jj0j4zsp0zw290eb70b9g26e
nginx: https://vccn7/api/reporting/jobs/download/jj0j4zsp0zw290eb70b9g26e
LB: https://metrics.xyz.com/api/reporting/jobs/download/jj0j4zsp0zw290eb70b9g26e

However it's still not working.

18

Hey,

so this indicates, that the cert you are using in kibana is not known on the elasticsearch side of things. Is this possible?

You can also configure an own truststore for the httpclient that is used by watcher to connect to kibana for example.

xpack.http.ssl.truststore.path: ...

--Alex

19

I had added the truststore on the coordinator nodes only, but noticed i the logs, that the watches were being run randomly from the data nodes or coordinator nodes. So I added the truststore to all of them, and this has solved the issue. Thanks!

20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.