Xpack, slack, elastic5.6, and certificates

[jamesliu_london]

I've setup SlackNotifications on my ES5.6/X-Pack (RHEL7.3). But I see this alert:

[2018-01-03T14:59:44,427][ERROR][o.e.x.n.s.SlackService ] [XXXXXXXXXXXX] failed to execute slack api http request
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) ~[?:?]
[...]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397) ~[?:?]

=============================================================

My Process shows it's using JDK1.8 and I've hardcoded the cacert-trustkey.

elastic+ 18155 1 78 14:46 ? 00:14:30 /usr/lib/jvm/jdk1.8.0_151/bin/java -Xms8g -Xmx8g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+DisableExplicitGC -XX:+AlwaysPreTouch -server -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -Djdk.io.permissionsUseCanonicalPath=true -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Dlog4j.skipJansi=true -Djavax.net.debug=ssl,handshake,record -Djavax.net.ssl.trustStore=/usr/lib/jvm/jdk1.8.0_151/jre/lib/security/cacerts -Djavax.net.ssl.trustStorePassword=XXXXXXXX -XX:+HeapDumpOnOutOfMemoryError -Des.path.home=/usr/share/elasticsearch -cp /usr/share/elasticsearch/lib/elasticsearch-5.3.0.jar:/usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -p /var/run/elasticsearch/elasticsearch.pid --quiet -Edefault.path.logs=/var/log/elasticsearch -Edefault.path.data=/var/lib/elasticsearch -Edefault.path.conf=/etc/elasticsearch

=============================================================

my ES config is

cluster.name: DDDDDDD
node.name: lllllllllllll
path.data: /opt/elastic
network.host: AAAAAAAAAA
http.port: 9200
discovery.zen.ping.unicast.hosts: ["AAAAAAAAA", "BBBBBBBBB", "CCCCCCCC"]
discovery.zen.minimum_master_nodes: 2

xpack.security.audit.enabled: true
xpack.security.audit.outputs: [logfile, index]
xpack.security.audit.index.settings:
index:
number_of_shards: 1
number_of_replicas: 1
#SSL Configs BEGIN ANSIBLE MANAGED BLOCK
xpack.ssl.key: /etc/elasticsearch/XXXXXXXXXXXXXX/XXXXXXXXXXXXXX.key
xpack.ssl.certificate: /etc/elasticsearch/XXXXXXXXXXXXXX/XXXXXXXXXXXXXX.crt
xpack.ssl.certificate_authorities: ["/etc/elasticsearch/XXXX-chain.crt"]
xpack.security.transport.ssl.enabled: false
xpack.security.http.ssl.enabled: false
xpack.security.authc:
realms:
native:
type: native
order: 0
file:
type: file
order: 1
#SSL Configs END ANSIBLE MANAGED BLOCK

xpack.notification.slack:
account:
monitoring:
url: https://hooks.slack.com/services/TTTTTTTTT/BBBBBBBBB/LLLLLLLLLLLLLLLLLLLL
[root@XXXXXXXXXXXX elasticsearch]#

==============================================================

Investigations I've made

• the server-subnet doesn't appear to be behind an interception-proxy (but I have added root-CA for slack to both):

  • /etc/elasticsearch/XXXX-chain.crt
  • /usr/lib/jvm/jdk1.8.0_151/jre/libs/security/cacerts

• (also - but not the main issue - which RootCA declaration is it using during a notification - (a) $JAVA_HOME/jre/lib/security/cacert or (b) xpack.ssl.certificate_authorities when the logs talk about "valid certification path to requested target)"

• X-pack is installed
  /usr/share/elasticsearch/bin/elasticsearch-plugin list
  x-pack

[ikakavas (Ioannis Kakavas)]

Hello,

The current behavior is that when xpack.ssl.certificate_authorities is set as you have in your elasticsearch.yml file, Watcher uses that in order to perform the certificate chain validation.
In fact , any of the watcher TLS/SSL setting you do not explicitly set will take its value from the default (as stated in the docs)

You need to allow Watcher to trust both the local CA so that it can connect to the local Elasticsearch cluster and the CAs your system trusts in order to be able to connect to hooks.slack.com over https . One way to solve this would be to add your local CA certificate to the system truststore and point your configuration to that.

By the way, I see that you have TLS/SSL disabled

xpack.security.transport.ssl.enabled: false
xpack.security.http.ssl.enabled: false

if this is the case, you wouldn't need to set xpack.ssl.certificate_authorities: ["/etc/elasticsearch/XXXX-chain.crt"] either, which will make Watched use the system truststore and thus trust the certificate of hooks.slack.com