Nginx in front of ES / SSL cert issue - ELK watcher webhook to enable / disable watcher fails

[adam.klein]

Running Elastic 7.8.0 locally, three elastic, two kibana, two logstash.

One nginx load-balancer in front of the two kibana nodes and another nginx load-balancer in front of the three elastic nodes.

Setting up a custom watcher that checks for healthcheck on a Hashicorp consul node, one watcher for up, another for down.

A couple of issues, most frustrating first:

1. The up watcher has a webhook that does a _deactivate on itself when it is triggered (so it doesn't keep sending "the service is responding and everything is fine" alerts. The down watcher has a webhook that does an _activate on the up watcher when it is triggered, so we are notified when the service comes back up. The _deactivate webhook is consistently failing with

    "status": "failure",
            "error": {
              "root_cause": [
                {
                  "type": "s_s_l_handshake_exception",
                  "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
                }

The watchers are both configured to hit the nginx balancer URL with the watcher path for the webhooks. I have confirmed the SSL certificate on the nginx balancer has the full certificate chain AND that the certificate on each of the elasticsearch nodes has the full chain. I have read that I should add the cert chain to the elasticsearch keystore, but cannot find any documentation or clear instruction on how to do this

2. The up watcher send_email notification works properly. The down watcher send_email does not. The configurations are the same except for the body of the message. The state of the down notification is always "awaits_successful_execution". I don't know what that means.

Any help would be appreciated. Thanks!

Adam Klein

[TimV (Tim Vernum)]

https://www.elastic.co/guide/en/elasticsearch/reference/7.8/notification-settings.html#ssl-notification-settings

It sounds like you need to configure your ES nodes to trust the Issuing CA for the webhook - that is, some part of the certificate chain for the whatever is serving up the Consul node (consul itself, or a load balancer)

[adam.klein]

Tim,

Thank you VERY much. Setting xpack.http.ssl.verification_mode to none unblocked the "up" watcher from disabling itself. I really do not understand why, if all the elasticsearch nodes and the nginx node fronting the elasticsearch nodes all have the full cert chain, and the cert with the full chain is pointed to by xpack.security.http.ssl.certificate_authorities in my elasticsearch.yml it would still require that... anyway. THANK YOU!

Next question. My "down" watcher is consistently reporting for all actions (email, notify our alerting system, and activate the "up" watcher) a status of "state": "awaits_successful_execution"

I've been researching and all I get is you will see that state if you're throttling, but, I'm NOT throttling.

Thanks again.

Adam Klein