XPack Watcher - SSL Error on Slack action

[aaronloesattv (Aaron L)]

I'm trying out the slack integration with watcher on our monitoring cluster. sending to slack is failing with the below exception.

"sent_messages": [
        {
          "status": "failure",
          "error": {
            "root_cause": [
              {
                "type": "s_s_l_handshake_exception",
                "reason": "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
              }
            ],
            "type": "s_s_l_handshake_exception",
            "reason": "sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
            "caused_by": {
              "type": "validator_exception",
              "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target",
              "caused_by": {
                "type": "sun_cert_path_builder_exception",
                "reason": "unable to find valid certification path to requested target"
              }
            }
          },
          "to": "#monitoring-production",
          "message": {
            "from": "_inlined_",
            "text": "this is a test. please ignore"
          }
        }
      ]

We are using Elasticsearch 6.1.3 docker image. I've verified that the GeoTrust Global CA fingerprint is located in the Java cert store stored at /etc/pki/ca-trust/extracted/java on the docker image (last one listed is the same fingerprint as the root CA).

[root@bab2f403b7f7 java]# keytool --keystore cacerts --storepass changeit --list | grep -A1 geo
geotrustprimarycertificationauthority, Jan 7, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 32:3C:11:8E:1B:F7:B8:B6:52:54:E2:E2:10:0D:D6:02:90:37:F0:96
--
geotrustuniversalca2, Jan 7, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 37:9A:19:7B:41:85:45:35:0C:A6:03:69:F3:3C:2E:AF:47:4F:20:79
--
geotrustprimarycertificationauthority-g3, Jan 7, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 03:9E:ED:B8:0B:E7:A0:3C:69:53:89:3B:20:D2:D9:32:3A:4C:2A:FD
geotrustprimarycertificationauthority-g2, Jan 7, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): 8D:17:84:D5:37:F3:03:7D:EC:70:FE:57:8B:51:9A:99:E6:10:D7:B0
--
geotrustglobalca2, Jan 7, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): A9:E9:78:08:14:37:58:88:F2:05:19:B0:6D:2B:0D:2B:60:16:90:7D
--
geotrustuniversalca, Jan 7, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): E6:21:F3:35:43:79:05:9A:4B:68:30:9D:8A:2F:74:22:15:87:EC:79
--
geotrustglobalca, Jan 7, 2018, trustedCertEntry, 
Certificate fingerprint (SHA1): DE:28:F4:A4:FF:E5:B9:2F:A3:C5:03:D1:A3:49:A7:F9:96:2A:82:12

in our cluster, we do set xpack.ssl.key, xpack.ssl.certificate, and xpack.ssl.certificate_authorities with values for our own self signed certificates for use with transport and http security. are these possibly getting in the way of watcher using the java trusted certificates? if so, should i be using the more specific settings for transport and http to set our self signed certs so that watcher doesn't use them exclusively?

[aaronloesattv (Aaron L)]

i'm just noticing that watcher uses the xpack.http.ssl.* properties and not its own. So in our case, since we have self signed certificates for transport and http, would the following resolve the issue (assuming that setting xpack.ssl.certificate_authorities is overriding the default list of CA's that would be trusted and not just appending to the list)

xpack.ssl.key - private key of elasticsearch node
xpack.ssl.certificate - public certificate of elasticsearch node
xpack.transport.ssl.certificate_authorities - to certificate chain of intermediate and root internal CA

this way xpack.http.ssl.certificate_authorities remains at the default value.

[aaronloesattv (Aaron L)]

this appears to be the case. i was able to reproduce the issue on a local docker cluster. the setting xpack.ssl.certificate_authorities is getting in the way of the slack integrations ability to call out to slack. by changing the property to xpack.security.transport.ssl.certificate_authorities, my cluster still operates normally and slack integration now works.

i'll wait to mark this reply as the answer as i want to allow some time for other answers/confirmations.