Watcher is not generating Email during anomaly

Elastic Stack Elasticsearch
1

Hi,
I am using Single metric ML job to detect an anomaly in my logs. I have also configured WATCHER TO generate EMAIL during any critical anomaly.
But I found that the single metric viewer is showing 1 critical anomaly but I did not receive an email alert for that.
In the ES logs I am receiving this message continuously-

[2018-08-23T11:36:29,088][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:38:14,162][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:39:59,193][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:41:43,922][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:43:29,309][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:45:13,838][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:46:58,878][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:48:43,906][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:50:28,935][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:52:13,966][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:53:59,035][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:55:44,070][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:57:29,097][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T11:59:14,150][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T12:00:59,191][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]
[2018-08-23T12:02:44,221][ERROR][o.e.x.w.i.s.ExecutableSearchInput] [node1] failed to execute [search] input for watch [ml-job-iteration14-20aug], reason [cannot write xcontent for unknown value of type class java.time.LocalDateTime]

Is this the reason that I didnt receive email of critical watcher...??
if no then what is the issue.

During threshold watcher setting I successfully sent the sample email through watcher.

2

Hey,

yes that is very likely the reason. This is a bug in Elasticsearch. You can fix this for by modifying the watch and appending .toString() at the end of the scripted fields that are calling LocalDateTime.

See https://github.com/elastic/elasticsearch/issues/31853 and https://github.com/elastic/kibana/pull/21998

3

Hi Alex,
Thanks for the support. But where must I make the suggest change i.e where must I append .toString().
I simply applied the watcher on my ML job that is being run on access log.

4

See this for how to edit the watch and insert the .toString()