# Elastic Watcher Alert Failing (Memory & Cpu Usage) **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-alerting\ **Created:** [February 4, 2016, 3:30pm UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974 "2016-02-04T15:30:29Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![pgcr](https://avatars.discourse-cdn.com/v4/letter/p/bcef8e/32.png) [@pgcr](https://discuss.elastic.co/u/pgcr)\ **Post date:** [February 4, 2016, 3:30pm UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974/1 "2016-02-04T15:30:29Z")
I have been attempting to setup a watch using the template that can be found here: [https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html](https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html) Current setup: 1 master node with 2 backups; 3 data nodes (3 shards for each); 1 clientnode. Elasticsearch is at 2.1.1, fluentd 2.3.0, watcher plugin latest ver, marvel plugin latest ver, license also installed (no account). This setup is running on CentOs on AWS. The two alerts I am setting up is High CPU usage and high jvm memory usage, for testing purposes I have set the alert to notify me if they are above 3% with an interval of 10s. Using plugin/head I am able to determine that these are in fact running every 10s, but normally I receive execution\_not\_needed or failed. When checking the log under condition I see: "condition": { "type": "script", "status": "failure", "reason": "GroovyScriptExecutionException[failed to run inline script [if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value \>= 3;] using lang [groovy]]; nested: NullPointerException[Cannot get property 'minutes' on null object]; " }, I have script.inline: on script.indexed: on On all data nodes and master node. Any help & information is greatly appreciated. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 8, 2016, 8:22am UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974/2 "2016-02-08T08:22:07Z")
Hey, can you use the [execute Watch API](https://www.elastic.co/guide/en/watcher/current/api-rest.html#api-rest-execute-watch) and paste the output here? Thanks! --Alex ---
**Author:** ![pgcr](https://avatars.discourse-cdn.com/v4/letter/p/bcef8e/32.png) [@pgcr](https://discuss.elastic.co/u/pgcr)\ **Post date:** [February 8, 2016, 3:01pm UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974/3 "2016-02-08T15:01:37Z")
Hey @spinscale, After executing the watch, I receive the same error. { "\_id": "mem\_watch\_9-2016-02-08T14:58:55.046Z", "watch\_record": { "watch\_id": "mem\_watch", "state": "executed", "trigger\_event": { "type": "manual", "triggered\_time": "2016-02-08T14:58:55.037Z", "manual": { "schedule": { "scheduled\_time": "2016-02-08T14:58:55.045Z" } } }, "input": { "search": { "request": { "search\_type": "query\_then\_fetch", "indices": [ ".marvel-\*" ], "types": [], "body": { "size": 0, "query": { "filtered": { "filter": { "range": { "@timestamp": { "gte": "now-2m", "lte": "now" } } } } }, "aggs": { "minutes": { "date\_histogram": { "field": "@timestamp", "interval": "minute" }, "aggs": { "nodes": { "terms": { "field": "node.name.raw", "size": 10, "order": { "memory": "desc" } }, "aggs": { "memory": { "avg": { "field": "jvm.mem.heap\_used\_percent" } } } } } } } } } } }, "condition": { "script": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value \>= 3;" }, "messages": [], "result": { "execution\_time": "2016-02-08T14:58:55.046Z", "execution\_duration": 97, "input": { "type": "simple", "status": "success", "payload": { "foo": "bar" } }, "condition": { "type": "always", "status": "success", "met": true }, "actions": [ { "id": "send\_email", "type": "email", "status": "failure", "transform": { "type": "script", "status": "failure", "reason": "GroovyScriptExecutionException[failed to run inline script [def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.findAll { return it.memory && it.memory.value \>= 3 };] using lang [groovy]]; nested: NullPointerException[Cannot get property 'minutes' on null object]; " }, "reason": "Failed to transform payload" } ] } } } Regards, Petro ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 9, 2016, 10:34am UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974/4 "2016-02-09T10:34:48Z")
Hey, tested locally. You dont have any marvel data to check against (thats how I get this error reproduced). What happens here is, that the watch expects the aggregations data structure to be there, what only happens, if data has been indexed. Have you installed the `marvel-agent` and is it indexing into your local cluster? --Alex ---
**Author:** ![pgcr](https://avatars.discourse-cdn.com/v4/letter/p/bcef8e/32.png) [@pgcr](https://discuss.elastic.co/u/pgcr)\ **Post date:** [February 9, 2016, 8:10pm UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974/5 "2016-02-09T20:10:35Z")
Hey Alex, Yep, I have marvel-agent running on the cluster (double-checked). Using \_plugin/head/browser I can see the marvel files being generated... The only thing I can think of is that i have not re-indexed the files manually. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [February 10, 2016, 10:32am UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974/6 "2016-02-10T10:32:42Z")
Hey, something is wrong with your watch, it does not execute a search query. Check the `result` section of your pasted response, it shows a `simple` input... --Alex ---
**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\ **Post date:** [June 9, 2016, 9:56am UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974/7 "2016-06-09T09:56:38Z")
Hi Alex, I am also having similar type of issue with watcher. I don't receive any email for cpu and memory usage. I know my email configuration in elasticsearch.yml is correct because I receive email for another watch. I have followed [https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html#watching-cpu-usage](https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html#watching-cpu-usage) and set the cpu usage to 5% just to check if I receive any email. After reading this post I have checked POST \_watcher/watch/cpu\_usage/\_execute which shows me output like this... { "\_id": "cpu\_usage\_168-2016-06-09T09:44:12.366Z", "watch\_record": { "watch\_id": "cpu\_usage", "state": "execution\_not\_needed", "trigger\_event": { "type": "manual", "triggered\_time": "2016-06-09T09:44:12.366Z", "manual": { "schedule": { "scheduled\_time": "2016-06-09T09:44:12.366Z" ... .... ... I have checked in marvel that my node is consuming more than 10% cpu all the time. Still I don't receive any email. Do you have any solution for me? (I'm a beginner in Elasticsearch and everything..so detailed answer would be really appreciated) thanks in advance. --Iqbal ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [June 9, 2016, 10:31am UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974/8 "2016-06-09T10:31:33Z")
hey, please open a new thread and include the output of calling the `Execute Watch Api`. --Alex ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:45pm UTC](https://discuss.elastic.co/t/elastic-watcher-alert-failing-memory-cpu-usage/40974/9 "2017-07-06T13:45:05Z")