# Nginx in front of ES / SSL cert issue - ELK watcher webhook to enable / disable watcher fails

**URL:** <https://discuss.elastic.co/t/nginx-in-front-of-es-ssl-cert-issue-elk-watcher-webhook-to-enable-disable-watcher-fails/244640>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [August 11, 2020, 10:26pm UTC](https://discuss.elastic.co/t/nginx-in-front-of-es-ssl-cert-issue-elk-watcher-webhook-to-enable-disable-watcher-fails/244640 "2020-08-11T22:26:09Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![adam.klein](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@adam.klein](https://discuss.elastic.co/u/adam.klein)\
**Post date:** [August 11, 2020, 10:26pm UTC](https://discuss.elastic.co/t/nginx-in-front-of-es-ssl-cert-issue-elk-watcher-webhook-to-enable-disable-watcher-fails/244640/1 "2020-08-11T22:26:09Z")

</div>

Running Elastic 7.8.0 locally, three elastic, two kibana, two logstash.

One nginx load-balancer in front of the two kibana nodes and another nginx load-balancer in front of the three elastic nodes.

Setting up a custom watcher that checks for healthcheck on a Hashicorp consul node, one watcher for up, another for down.

A couple of issues, most frustrating first:

1. The up watcher has a webhook that does a \_deactivate on itself when it is triggered (so it doesn't keep sending "the service is responding and everything is fine" alerts. The down watcher has a webhook that does an \_activate on the up watcher when it is triggered, so we are notified when the service comes back up. The \_deactivate webhook is consistently failing with

```auto
    "status": "failure",
            "error": {
              "root_cause": [
                {
                  "type": "s_s_l_handshake_exception",
                  "reason": "PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
                }

```

The watchers are both configured to hit the nginx balancer URL with the watcher path for the webhooks. I have confirmed the SSL certificate on the nginx balancer has the full certificate chain AND that the certificate on each of the elasticsearch nodes has the full chain. I have read that I should add the cert chain to the elasticsearch keystore, but cannot find any documentation or clear instruction on how to do this

1. The up watcher send\_email notification works properly. The down watcher send\_email does not. The configurations are the same except for the body of the message. The state of the down notification is _always_ "awaits\_successful\_execution". I don't know what that means.

Any help would be appreciated. Thanks!

Adam Klein

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [August 12, 2020, 1:19am UTC](https://discuss.elastic.co/t/nginx-in-front-of-es-ssl-cert-issue-elk-watcher-webhook-to-enable-disable-watcher-fails/244640/2 "2020-08-12T01:19:14Z")

</div>

[https://www.elastic.co/guide/en/elasticsearch/reference/7.8/notification-settings.html#ssl-notification-settings](https://www.elastic.co/guide/en/elasticsearch/reference/7.8/notification-settings.html#ssl-notification-settings)

It sounds like you need to configure your ES nodes to trust the Issuing CA for the webhook - that is, some part of the certificate chain for the whatever is serving up the Consul node (consul itself, or a load balancer)

---

<div class="post-metadata">

**Author:** ![adam.klein](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@adam.klein](https://discuss.elastic.co/u/adam.klein)\
**Post date:** [August 12, 2020, 8:48pm UTC](https://discuss.elastic.co/t/nginx-in-front-of-es-ssl-cert-issue-elk-watcher-webhook-to-enable-disable-watcher-fails/244640/3 "2020-08-12T20:48:06Z")

</div>

Tim,

Thank you _VERY_ much. Setting `xpack.http.ssl.verification_mode` to `none` unblocked the "up" watcher from disabling itself. I really do not understand why, if all the elasticsearch nodes and the nginx node fronting the elasticsearch nodes _all_ have the full cert chain, and the cert with the full chain is pointed to by `xpack.security.http.ssl.certificate_authorities` in my `elasticsearch.yml` it would still require that... anyway. THANK YOU!

Next question. My "down" watcher is consistently reporting for all actions (email, notify our alerting system, and activate the "up" watcher) a status of `"state": "awaits_successful_execution"`

I've been researching and all I get is you will see that state if you're throttling, but, I'm NOT throttling.

Thanks again.

Adam Klein

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 9, 2020, 8:48pm UTC](https://discuss.elastic.co/t/nginx-in-front-of-es-ssl-cert-issue-elk-watcher-webhook-to-enable-disable-watcher-fails/244640/4 "2020-09-09T20:48:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.