# Not receiving email for CPU usage

**URL:** <https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [June 9, 2016, 10:43am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316 "2016-06-09T10:43:33Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 9, 2016, 10:43am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/1 "2016-06-09T10:43:33Z")

</div>

I am with watcher. I don't receive any email for cpu and memory usage. I know my email configuration in  
elasticsearch.yml is correct because I receive email for another watch (i.e. event\_critical\_watch). I have followed https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html#watching-cpu-usage and set the cpu usage to 5% just to check if I receive any email. After reading another post like this in this forum, I have checked `POST _watcher/watch/cpu_usage/_execute` which shows me output like this...(in the 1st comment)

I have checked in marvel that my node is consuming more than 10% cpu all  
the time. Still I don't receive any email. Does any one have any solution for  
me?

Just FYI: I have checked in kibana .marvel-\* index but there I didn't find any "os.cpu.user" field as mentioned in the guide. Is it the possible reason? If yes, why that field is not there?

(I'm a beginner in Elasticsearch and everything..so detailed answer  
would be really appreciated)  
thanks in advance.  
--Iqbal

---

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 9, 2016, 10:45am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/2 "2016-06-09T10:45:22Z")

</div>

{  
"\_id": "cpu\_usage\_220-2016-06-09T10:35:15.687Z",  
"watch\_record": {  
"watch\_id": "cpu\_usage",  
"state": "execution\_not\_needed",  
"trigger\_event": {  
"type": "manual",  
"triggered\_time": "2016-06-09T10:35:15.687Z",  
"manual": {  
"schedule": {  
"scheduled\_time": "2016-06-09T10:35:15.687Z"  
}  
}  
},  
"input": {  
"search": {  
"request": {  
"search\_type": "query\_then\_fetch",  
"indices": [  
".marvel-_"  
],  
"types": [],  
"body": {  
"size": 0,  
"query": {  
"filtered": {  
"filter": {  
"range": {  
"@timestamp": {  
"gte": "now-2m",  
"lte": "now"  
}  
}  
}  
}  
},  
"aggs": {  
"minutes": {  
"date\_histogram": {  
"field": "@timestamp",  
"interval": "minute"  
},  
"aggs": {  
"nodes": {  
"terms": {  
"field": "node.name.raw",  
"size": 10,  
"order": {  
"cpu": "desc"  
}  
},  
"aggs": {  
"cpu": {  
"avg": {  
"field": "os.cpu.user"  
}  
}  
}  
}  
}  
}  
}  
}  
}  
}  
},  
"condition": {  
"script": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.cpu && node.cpu.value \>= 5;"  
},  
"messages": [],  
"result": {  
"execution\_time": "2016-06-09T10:35:15.687Z",  
"execution\_duration": 1,  
"input": {  
"type": "search",  
"status": "success",  
"payload": {  
"\_shards": {  
"total": 2,  
"failed": 0,  
"successful": 2  
},  
"hits": {  
"hits": [],  
"total": 0,  
"max\_score": 0  
},  
"took": 1,  
"timed\_out": false,  
"aggregations": {  
"minutes": {  
"buckets": []  
}  
}  
},  
"search": {  
"request": {  
"search\_type": "query\_then\_fetch",  
"indices": [  
".marvel-_"  
],  
"types": [],  
"template": {  
"template": {  
"size": 0,  
"query": {  
"filtered": {  
"filter": {  
"range": {  
"@timestamp": {  
"gte": "now-2m",  
"lte": "now"  
}  
}  
}  
}  
},  
"aggs": {  
"minutes": {  
"date\_histogram": {  
"field": "@timestamp",  
"interval": "minute"  
},  
"aggs": {  
"nodes": {  
"terms": {  
"field": "node.name.raw",  
"size": 10,  
"order": {  
"cpu": "desc"  
}  
},  
"aggs": {  
"cpu": {  
"avg": {  
"field": "os.cpu.user"  
}  
}  
}  
}  
}  
}  
}  
},  
"params": {  
"ctx": {  
"metadata": null,  
"watch\_id": "cpu\_usage",  
"id": "cpu\_usage\_220-2016-06-09T10:35:15.687Z",  
"trigger": {  
"triggered\_time": "2016-06-09T10:35:15.687Z",  
"scheduled\_time": "2016-06-09T10:35:15.687Z"  
},  
"vars": {},  
"execution\_time": "2016-06-09T10:35:15.687Z"  
}  
}  
}  
}  
}  
},  
"condition": {  
"type": "script",  
"status": "success",  
"met": false  
},  
"actions": []  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 9, 2016, 11:25am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/3 "2016-06-09T11:25:39Z")

</div>

Hey

The last five lines tell you the important part:

```auto
...
"condition": {
"type": "script",
"status": "success",
"met": false
},
...

```

This means that the condition returned false. You should go back and evaluate the condition more closely. Maybe you referenced a wrong path somewhere?

--Alex

---

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 9, 2016, 12:30pm UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/4 "2016-06-09T12:30:54Z")

</div>

Hi..  
Thanks for the quick reply. My condition field is the same as mentioned in the link ([https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html#watching-cpu-usage](https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html#watching-cpu-usage)). I have just changed 75 to 5 to check if I receive email.

"condition": {  
"script": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.cpu && node.cpu.value \>= 5;"  
},

what could be the mistake here? should I adjust anything according to my settings in the elasticsearch. I have only one node called 'My 1st node'  
...  
Iqbal

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 9, 2016, 12:42pm UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/5 "2016-06-09T12:42:32Z")

</div>

Hey,

please take your time and examine the result of the execute watch API. Check the `result`, which contains the search response and find out no hits at all a returned and the `buckets` array is empty. So either you are querying the wrong index or the index does not exist on your local cluster.

--Alex

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 9, 2016, 1:02pm UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/6 "2016-06-09T13:02:37Z")

</div>

Hey,

I took a look at the example, and I think it does not reflect the current marvel stats.

- Can you replace the two occurences of `@timestamp` with `timestamp`.
- Can you replace the mention of `node.name.raw` with `node.name`
- Can you change the heap percent mention from `jvm.mem.heap_used_percent` to `node_stats.jvm.mem.heap_used_percent`
- Last but not least, can you add `"types":["node_stats"]` after the `indices` part to configure the correct query of the type?

Let's see if that changes anything!

--Alex

---

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 9, 2016, 3:07pm UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/7 "2016-06-09T15:07:12Z")

</div>

Hi Alex,

`jvm.mem.heap_used_percent` is not there in CPU usage, rather in [Memory usage](https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html#watching-memory-usage) example. Still I tried with the memory usage example. Changed the terms as you suggested. But there is no change in the execute result:

> ```
> }
> },
> "condition": {
> "type": "script",
> "status": "success",
> "met": false
> },
> "actions": []
> }
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 10, 2016, 7:12am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/8 "2016-06-10T07:12:30Z")

</div>

Hey,

can you provide the full watch you are testing with. Also, please put it in appropriate formatting tags, see [here](http://commonmark.org/help/) how to use use code blocks. This makes it much easier for others.

--Alex

---

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 10, 2016, 8:43am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/9 "2016-06-10T08:43:21Z")

</div>

Hi,  
please find my complete watch below:

```
  PUT _watcher/watch/mem_watch
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          ".marvel-*"
        ],
        "types": [
          "node_stats"
        ],
        "body": {
          "size": 0,
          "query": {
            "filtered": {
              "filter": {
                "range": {
                  "timestamp": {
                    "gte": "now-2m",
                    "lte": "now"
                  }
                }
              }
            }
          },
          "aggs": {
            "minutes": {
              "date_histogram": {
                "field": "timestamp",
                "interval": "minute"
              },
              "aggs": {
                "nodes": {
                  "terms": {
                    "field": "node.name",
                    "size": 10,
                    "order": {
                      "memory": "desc"
                    }
                  },
                  "aggs": {
                    "memory": {
                      "avg": {
                        "field": "node_stats.jvm.mem.heap_used_percent"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "throttle_period": "2m",
  "condition": {
    "script": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value >= 5;"
  },
  "actions": {
    "send_email": {
      "transform": {
        "script": "def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.findAll { return it.memory && it.memory.value >=5 };"
      },
      "email": {
        "to": "user@mycompany.com",
        "subject": "Watcher Notification - HIGH MEMORY USAGE",
        "body": "Nodes with HIGH MEMORY Usage (above 5%):\n\n{{#ctx.payload._value}}\"{{key}}\" - Memory Usage is at {{memory.value}}%\n{{/ctx.payload._value}}"
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 10, 2016, 3:15pm UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/10 "2016-06-10T15:15:44Z")

</div>

Hey,

the field for the `terms` agg must be `source_node.name` instead of `node.name`, my fault.

--Alex

---

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 13, 2016, 8:05am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/11 "2016-06-13T08:05:33Z")

</div>

Hi Alex,  
It has worked 😃 Thanks a lot.  
Is there any way to edit a watch?  
To make a small change, I have to DELETE and PUT again to get `"created": true`

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 13, 2016, 8:09am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/12 "2016-06-13T08:09:43Z")

</div>

Hey,

just put it again, it's fine. The watch will be overwritten.

--Alex

---

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 13, 2016, 8:25am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/13 "2016-06-13T08:25:18Z")

</div>

```
Hi,
Thanks again.
Now could you please review my cpu_usage watch? I'm not receiving any email for high cpu usage. Here is my watch for that: 

        PUT _watcher/watch/cpu_usage
        {
          "trigger": {
            "schedule": {
              "interval": "1m"
            }
          },
          "input": {
            "search": {
              "request": {
                "indices": 
                "types":["node_stats"]
                [
                  ".marvel-*"
                ],
                "body": {
                  "size" : 0,
                  "query": {
                    "filtered": {
                      "filter": {
                        "range": {
                          "timestamp": {
                            "gte": "now-2m",
                            "lte": "now"
                          }
                        }
                      }
                    }
                  },
                  "aggs": {
                    "minutes": {
                      "date_histogram": {
                        "field": "timestamp",
                        "interval": "minute"
                      },
                      "aggs": {
                        "nodes": {
                          "terms": {
                            "field": "source_node.name",
                            "size": 10,
                            "order": {
                              "cpu": "desc"
                            }
                          },
                          "aggs": {
                            "cpu": {
                              "avg": {
                                "field": "os.cpu.user"
                              }
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          },
          "throttle_period": "1m", 
          "condition": {
            "script": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.cpu && node.cpu.value >= 5;"
          },
          "actions": {
            "send_email": { 
              "transform": {
                "script": "def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.findAll { return it.cpu && it.cpu.value >= 5 };"
              },
              "email": {
                "to": "user@mycompany.com", 
                "subject": "Watcher Notification - HIGH CPU USAGE",
                "body": "Nodes with HIGH CPU Usage (above 5%):\n\n{{#ctx.payload._value}}\"{{key}}\" - CPU Usage is at {{cpu.value}}%\n{{/ctx.payload._value}}"
              }
            }
          }
        }
```

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 14, 2016, 7:08am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/14 "2016-06-14T07:08:41Z")

</div>

Hey,

please execute the query standalone first and see if you get back any buckets. If not, execute a search and see where the documents differ.

--Alex

---

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 14, 2016, 8:34am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/15 "2016-06-14T08:34:50Z")

</div>

Hi Alex,

I have done `POST _watcher/watch/cpu_usage/_execute` and I think I haven't got any buckets.

```
   "buckets": [
                      {
                        "doc_count": 4,
                        "cpu": {
                          "value": null

```

Then, I have also compared between mem\_watch and cpu\_usage. I have changed from `os.cpu.user`to `node_stats.os.cpu.user` with no success so far. I am not an expert and maybe that's why I'm missing something.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 14, 2016, 9:12am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/16 "2016-06-14T09:12:14Z")

</div>

Hey,

this is not what I meant. You should execute the search operation that you refer to in the watch manually as well as a search operation by index and type manually (without specifying any queries) - just do `GET /.marvel-/node_stats/_search` - this allows you to check out if the fields you are referring to in your query are set in the returned documents.

--Alex

---

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 14, 2016, 9:54am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/17 "2016-06-14T09:54:59Z")

</div>

Hi,  
sorry for my ignorance. I have done `GET /.marvel-/node_stats/_search` but received `404`then I did `GET /.marvel-*/node_stats/_search` (please note `*`) which returned a lot of results but it doesn't contain any `node_stats.os.cpu.user`field  
Thanks.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 15, 2016, 8:13am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/18 "2016-06-15T08:13:54Z")

</div>

Hey,

if you execute

```json
GET /.marvel-es-*/node_stats/_search
{
  "size" : 1,
  "sort" : [{ "timestamp" : "desc" }]
}

```

You can see the JSON structure for the last node\_stats. The CPU load is now part of the `process` JSON being returned and only covers the load of this process and not the whole OS, which is currently not being monitored.

I will update the watches in the docs over the next days, but until then you can fix your watch by adapting to the JSON returned.

--Alex

---

<div class="post-metadata">

**Author:** ![iqbal\_nazir](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/iqbal_nazir/32/10216_2.png) [@iqbal\_nazir](https://discuss.elastic.co/u/iqbal_nazir)\
**Post date:** [June 15, 2016, 9:22am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/19 "2016-06-15T09:22:18Z")

</div>

Hi again,

Thanks for the clarification. Now could you please tell me how to adjust the watch. After executing your search I have found CPU percent is under `process` of the last `node_stats`. Then I have changed the field from `node_stats.os.cpu.user` to `node_stats.process.cpu.percent` with no success. Should I have to change anything else in the watch?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [June 15, 2016, 10:18am UTC](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316/20 "2016-06-15T10:18:20Z")

</div>

This watch works for me

```json
PUT _watcher/watch/cpu_usage
{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          ".marvel-es-1-*"
        ],
        "types" : [
          "node_stats"
        ],
        "body": {
          "size" : 0,
          "query": {
            "filtered": {
              "filter": {
                "range": {
                  "timestamp": {
                    "gte": "now-2m",
                    "lte": "now"
                  }
                }
              }
            }
          },
          "aggs": {
            "minutes": {
              "date_histogram": {
                "field": "timestamp",
                "interval": "minute"
              },
              "aggs": {
                "nodes": {
                  "terms": {
                    "field": "source_node.name",
                    "size": 10,
                    "order": {
                      "cpu": "desc"
                    }
                  },
                  "aggs": {
                    "cpu": {
                      "avg": {
                        "field": "node_stats.process.cpu.percent"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "throttle_period": "30m", <1>
  "condition": {
    "script": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.cpu && node.cpu.value >= 75;"
  },
  "actions": {
    "send_email": { <2>
      "transform": {
        "script": "def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.findAll { return it.cpu && it.cpu.value >= 75 };"
      },
      "email": {
        "to": "user@example.com", <3>
        "subject": "Watcher Notification - HIGH CPU USAGE",
        "body": "Nodes with HIGH CPU Usage (above 75%):\n\n{{#ctx.payload._value}}\"{{key}}\" - CPU Usage is at {{cpu.value}}%\n{{/ctx.payload._value}}"
      }
    }
  }
}

```

--Alex

[Next page](https://discuss.elastic.co/t/not-receiving-email-for-cpu-usage/52316.md?page=2)