Watcher for Reporting shows an SSL Exception in the output

swathi · June 30, 2018, 2:48am

Hello,

I am trying to create a watcher which generates daily/weekly reports of a dashboard and send that via emails. When the watcher is getting executed, it shows an SSL exception in the output (shown in the picture)

Below is the watcher I am using for this:

{
  "trigger": {
    "schedule": {
      "interval": "30s"
    }
  },
  "input": {
    "none": {}
  },
  "condition": {
    "always": {}
  },
  "actions": {
    "email_admin": {
      "email": {
        "profile": "standard",
        "attachments": {
          "error.pdf": {
            "reporting": {
              "url": "https://xyz",
              "retries": 5,
              "interval": "15s"
            }
          }
        },
        "to": [
          "myemail@mydomain.com"
        ],
        "subject": "Weekly Monitoring Report"
      }
    }
  }
}

Please help me understand why this error is occurring here and how to solve this. Thanks in advance.

ikakavas · July 2, 2018, 11:38am

Hi,

The problem happens when watcher attempts to download the file from https://xyz as it cannot validate the TLS certificate that is presented for https://xyz

I am assuming the URL is pointing to your Kibana instance. What are your SSL settings in Kibana? Are you using a self signed certificate for TLS in Kibana ? ( Step 1 in this https://www.elastic.co/guide/en/kibana/6.3/configuring-tls.html#configuring-tls )

swathi · July 3, 2018, 4:59pm

@ikakavas no, I am not using any self-signed certificate. Could that cause this problem?

ikakavas · July 3, 2018, 5:26pm

No, not necessarily, I was just probing for what your setup looks like since that was not included in the original information.

The issue is as I describe above, and the generic solution is to make sure that watcher trusts your certificate or the certificate authority that has signed it . It is sadly not possible to give you a specific suggestion without knowing the rest of your configuration but note that Watcher ssl settings with regards to what certificates are trusted are described here https://www.elastic.co/guide/en/elasticsearch/reference/current/notification-settings.html#_watcher_tls_ssl_key_and_trusted_certificate_settings . If you haven't explicitly set that then it falls back to your node's SSL settings ( xpack.ssl.X) . If those haven't been set either, then it falls back to the truststore of your JVM ( the cacert file that ships with your java installation ).

If you can share some more details regarding

What kind of a certificate you're using for kibana ( you said it's not self signed ,but is it signed by a public CA or an internal one ? ) I'd you don't know and you have access to openssl binary in your environment you can run

openssl s_client -connect xyz:443

and share the output, or at least your gained insights from running it.

Your SSL configuration in elasticsearch.yml

we would be able to offer some more specific suggestions

swathi · July 3, 2018, 5:44pm

Thanks for responding.

I am using a public CA certificate.
My SSL configuration in elasticsearch.yml is below:

xpack.ssl.keystore.path: certs-path
xpack.ssl.truststore.path: certs-path
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.verification_mode: certificate

jspeer · July 3, 2018, 9:17pm

@ikakavas I'm helping @swathi on this issue so might be able to help out some. The are using public CA for the SSL cert to the kibana UI. We are using self-signed certs for the node to node communications.

Here is our kibana reporting settings(i've removed the key but it's defined) :

xpack.reporting.enabled: true
xpack.reporting.encryptionKey:
xpack.reporting.kibanaServer.port: 443
xpack.reporting.kibanaServer.protocol: https
xpack.reporting.kibanaServer.hostname: metrics.xyz.com

TimV · July 3, 2018, 11:49pm

This is the cause of your problem.
This tells X-Pack to only trust "certs-path" and not trust the JRE's defaults CAs.
It is almost always a mistake to configure xpack.ssl.truststore.path or xpack.ssl.certificate_authorities. (I accept that our docs used to recommend that you do this - they shouldn't have).

You should instead set your config to be:

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.keystore.path: certs-path
xpack.security.transport.ssl.truststore.path: certs-path
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: certs-path

You almost certainly don't want to have this setting:

xpack.security.http.ssl.verification_mode: certificate

The settings below are all related to trust, and for HTTP that means they affect how PKI (client certificates) work, but don't affect the way SSL works for the server itself. Unless you are trying to configure a particular setup for PKI, you probably don't want to configure them for HTTP (but you typically do want them for transport).

verification_mode
truststore.path
certificate_authorities

jspeer · July 16, 2018, 6:01pm

OK, I've made these change to the config, however I'm still receiving the same Error when I try to execute the watch.

jspeer · July 16, 2018, 6:06pm

xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.keystore.path: certs-path xpack.security.transport.ssl.truststore.path: certs-path xpack.security.transport.ssl.verification_mode: certificate xpack.security.http.ssl.enabled: true xpack.security.http.ssl.keystore.path: certs-path

TimV · July 17, 2018, 7:18am

Do you have any more details on the exception - it's possible that it's coming from your SMTP server rather than Kibana. Are you accessing the email server over SSL?

spinscale · July 17, 2018, 7:22am

can you also please share the full output of the execute watch API or a full watch history entry? Thanks a lot!

jspeer · July 17, 2018, 3:22pm

This was my original thinking since the error was inside the email block of the script. However in other watcher's I am able to send simple emails, without attached reports. I've also verified that I'm not using SSL to the mail server.

jspeer · July 17, 2018, 3:27pm

Here is the output of a test execute. Thanks!

{
"watch_id": "_inlined_",
"node": "JBNgvNcpR_COY5YvJNncbQ",
"state": "executed",
"status": {
  "state": {
    "active": true,
    "timestamp": "2018-07-17T15:23:15.102Z"
  },
  "last_checked": "2018-07-17T15:23:15.103Z",
  "last_met_condition": "2018-07-17T15:23:15.103Z",
  "actions": {
    "email_admin": {
      "ack": {
        "timestamp": "2018-07-17T15:23:15.102Z",
        "state": "awaits_successful_execution"
      },
      "last_execution": {
        "timestamp": "2018-07-17T15:23:15.103Z",
        "successful": false,
        "reason": ""
      }
    }
},
"execution_state": "executed",
"version": -1
},
"trigger_event": {
"type": "manual",
"triggered_time": "2018-07-17T15:23:15.103Z",
"manual": {
  "schedule": {
    "scheduled_time": "2018-07-17T15:23:15.103Z"
  }
}
},
"input": {
"none": {}
 },
"condition": {
  "always": {}
},
"metadata": {
  "xpack": {
    "type": "json"
  }
},
"result": {
  "execution_time": "2018-07-17T15:23:15.103Z",
  "execution_duration": 7,
  "input": {
    "type": "none",
    "status": "success",
    "payload": {}
  },
  "condition": {
    "type": "always",
    "status": "success",
    "met": true
  },
  "actions": [
  {
    "id": "email_admin",
    "type": "email",
    "status": "failure",
    "error": {
      "root_cause": [
        {
          "type": "s_s_l_handshake_exception",
          "reason": "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
        }
      ],
      "type": "s_s_l_handshake_exception",
      "reason": "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
      "caused_by": {
        "type": "validator_exception",
        "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
        "caused_by": {
          "type": "sun_cert_path_builder_exception",
          "reason": "unable to find valid certification path to requested target"
        }
      }
    }
  }
]
},
"messages": []
}

spinscale · July 18, 2018, 7:54am

@jspeer as Tim already asked: Is it possible that you are using custom certificates with your mail server?

jspeer · July 18, 2018, 3:10pm

Here is my settings for the mail server:
xpack.notification.email.account:
smtp_account:
profile: standard
email_defaults:
from: svcs@xyz.com
smtp:
auth: false
starttls.enable: false
host: mailhost.xyz.com
port: 25

Also, I have emails being sent from other watches that are delivered properly. Wouldn't a misconfigured email certificate give the same error in that case?

spinscale · July 19, 2018, 7:18am

Hey,

indeed it would. Are you using custom certificates for TLS communication in Kibana as well? I assume that accessing kibana is causing this.

Can you remove the reporting attachment from that watch to verify if this is the case?

--Alex

jspeer · July 19, 2018, 4:57pm

I've removed the report section, and the email sends without a problem. So there is something about this portion:

"attachments": { "error.pdf": { "reporting": { "url": "https://vccn7:5601/api/reporting/jobs/download/jj0j4zsp0zw290eb70b9g26e", "retries": 5, "interval": "15s" } } },

There are a few certs in play here. The first is for node to node elasticsearch communication, and it is all self-signed. Next is our private CA signed cert for https connections via nginx or kibana. The last is for the public https URL metrics.xyz.com and it is signed by a public CA (this terminates on a load balancer).

I've replaced the URL inside the watcher with all 3 possiblities for the URL:
kibana: https://vccn7:5601/api/reporting/jobs/download/jj0j4zsp0zw290eb70b9g26e
nginx: https://vccn7/api/reporting/jobs/download/jj0j4zsp0zw290eb70b9g26e
LB: https://metrics.xyz.com/api/reporting/jobs/download/jj0j4zsp0zw290eb70b9g26e

However it's still not working.

spinscale · July 20, 2018, 8:12am

Hey,

so this indicates, that the cert you are using in kibana is not known on the elasticsearch side of things. Is this possible?

You can also configure an own truststore for the httpclient that is used by watcher to connect to kibana for example.

xpack.http.ssl.truststore.path: ...

--Alex

jspeer · August 13, 2018, 5:57pm

I had added the truststore on the coordinator nodes only, but noticed i the logs, that the watches were being run randomly from the data nodes or coordinator nodes. So I added the truststore to all of them, and this has solved the issue. Thanks!

system · September 10, 2018, 5:57pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.