# Some of built-in watches fails **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-alerting\ **Created:** [February 1, 2018, 8:20am UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960 "2018-02-01T08:20:15Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [February 1, 2018, 8:20am UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/1 "2018-02-01T08:20:15Z")
Hi, I have a situation where two of the built-in watches fails: Cluster status and Logstash version mismatch. Both Watcher in Kibana and Elasticsearch log file just states that it failed. I have also another issue with one of my own watches. I wanted to test out email action, so I created a dummy ML job and ticked the "send email" checkbox, copied the email config from that watch into another ML job. Now it just says that the email action Execution Failing, but there is no errors anywhere. Any pointers where to start? ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 7, 2018, 8:46am UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/2 "2018-02-07T08:46:17Z")
Hey Can you share the log file **and** the corresponding watch history entries please, so we can take a deeper look why those actually failed? Thank you! --Alex ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [February 7, 2018, 10:33am UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/3 "2018-02-07T10:33:36Z")
Hi! Thank you for replying. I have a feeling that this is related to email configuration, because it started right after I enabled it. But let's break it down: Kibana UI looks like this: ![Untitled%202](https://us1.discourse-cdn.com/elastic/original/3X/a/8/a8b40c1504201d2526994d2d48107a603226e9ab.png) ![Untitled%203](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a7fa005518c8cd400652d4aabd359c993ff6e23f.png) I have grepped through all the logs in all my 3 ES nodes. They have multiple events like this every day: > [2018-01-25T11:38:03,082][WARN][o.e.x.w.e.ExecutionService] [analyzer03] failed to execute watch [roClcHctTIWj8HFwteQHQw\_elasticsearch\_cluster\_status] > [2018-01-25T11:40:01,045][WARN][o.e.x.w.e.ExecutionService] [analyzer03] failed to execute watch [roClcHctTIWj8HFwteQHQw\_elasticsearch\_cluster\_status] > [2018-01-25T11:50:59,713][WARN][o.e.x.w.e.ExecutionService] [analyzer03] failed to execute watch [roClcHctTIWj8HFwteQHQw\_elasticsearch\_cluster\_status] > [2018-01-25T12:59:59,605][WARN][o.e.x.w.e.ExecutionService] [analyzer03] failed to execute watch [roClcHctTIWj8HFwteQHQw\_elasticsearch\_cluster\_status] > [2018-01-25T13:24:58,094][WARN][o.e.x.w.e.ExecutionService] [analyzer03] failed to execute watch [roClcHctTIWj8HFwteQHQw\_kibana\_version\_mismatch] > [2018-01-25T15:57:58,422][WARN][o.e.x.w.e.ExecutionService] [analyzer03] failed to execute watch [roClcHctTIWj8HFwteQHQw\_kibana\_version\_mismatch] > [2018-01-25T17:07:58,112][WARN][o.e.x.w.e.ExecutionService] [analyzer03] failed to execute watch [roClcHctTIWj8HFwteQHQw\_kibana\_version\_mismatch] The version mismatch is okay, it is 0.0.1 version back and forth, so it shouldn't really effect on this. I think this is related: > /var/log/elasticsearch/analyzer-prod-2018-01-25.log.gz:[2018-01-25T13:09:18,286][ERROR][o.e.x.w.a.e.ExecutableEmailAction] [analyzer01] failed to execute action [roClcHctTIWj8HFwteQHQw\_logstash\_version\_mismatch/send\_email\_to\_admin] > /var/log/elasticsearch/analyzer-prod-2018-01-25.log.gz:javax.mail.MessagingException: failed to send email with subject [[RESOLVED] X-Pack Monitoring: Logstash Version Mismatch (roClcHctTIWj8HFwteQHQw)] via account [exchange\_account] Doesn't say why it fails. If I query watcher-history from that same day, I found this: > GET .watcher-history-7-2018.01.25/\_search > { > "query" : { "match" : { "watch\_id": "roClcHctTIWj8HFwteQHQw\_elasticsearch\_cluster\_status" }} > } > ``` > "actions": { > "send_email_to_admin": { > "ack": { > "timestamp": "2017-12-19T08:10:25.512Z", > "state": "awaits_successful_execution" > }, > "last_execution": { > "timestamp": "2018-01-21T00:01:08.762Z", > "successful": false, > "reason": "" > } > > ``` The reason is empty. I browsed through the watch history and tried to find "error", "fail", "action", "email" but I couldn't find any error messages. I have configured email settings like this in elasticsearch.yml (on all nodes): > ``` > notification: > email: > html: > sanitization: > enabled: false > account: > exchange_account: > profile: outlook > email_defaults: > from: analyzer@company.tld > smtp: > starttls.enable: true > host: outlook.company.tld > port: 25 > > ``` If I try to connect to the mail server with cURL, it works from all the nodes: > [root@analyzer03 ~]# curl --ssl-reqd -v smtp://outlook.company.tld > > - About to connect() to outlook.company.tld port 25 (#0) > - Trying XXX.XXX.XXX.XXX... > - Connected to outlook.company.tld (XXX.XXX.XXX.XXX) port 25 (#0) > \< 220 exch2k16.intra.company.tld Microsoft ESMTP MAIL Service ready at Wed, 7 Feb 2018 12:27:16 +0200 > EHLO analyzer03 > \< 250-exch2k16.intra.company.tld Hello [XXX.XXX.XXX.XXX] > \< 250-SIZE 36700160 > \< 250-PIPELINING > \< 250-DSN > \< 250-ENHANCEDSTATUSCODES > \< 250-STARTTLS > \< 250-8BITMIME > \< 250-BINARYMIME > \< 250 CHUNKING > STARTTLS > \< 220 2.0.0 SMTP server ready Thanks! ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 7, 2018, 11:24am UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/4 "2018-02-07T11:24:01Z")
Hey, is there any possibility to see the full watch history entry? If it is too sensitive in public, you could also email it to me at $firstname.$lastname@elastic.co Indeed it seems as if there are problems sending your email, let's try to get down to the reason. --Alex ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [February 7, 2018, 1:06pm UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/5 "2018-02-07T13:06:06Z")
Hi! Just dropped you an email with watcher history attached. Cheers! ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 7, 2018, 1:57pm UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/6 "2018-02-07T13:57:26Z")
do you have any custom SSL setup? ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [February 7, 2018, 2:03pm UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/7 "2018-02-07T14:03:52Z")
Yes, the mail server uses a certificate signed by our internal CA. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 7, 2018, 2:06pm UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/8 "2018-02-07T14:06:40Z")
Does Elasticsearch have access to this internal CA cert? As it is needed when connecting... ---
**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\ **Post date:** [February 7, 2018, 2:08pm UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/9 "2018-02-07T14:08:19Z")
No, I have not configured it anywhere because I don't know how to. So is there a setting where I could define a path, or should I add it to system-wide certificates path, or... ? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 7, 2018, 2:08pm UTC](https://discuss.elastic.co/t/some-of-built-in-watches-fails/117960/10 "2018-03-07T14:08:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.