Memory Usage AND CPU Usage Not Work!

Elastic Stack Elasticsearch
1

Hi,
I'm following example in https://www.elastic.co/guide/en/watcher/current/watching-marvel-data.html#watching-memory-usage, I'm currently using elasticsearch, kibana and metricbeat version 5.4.1 setup on windows10. I created watcher memory usage and cpu usage i should get alert with email. When i executed the watch api, I'm getting this error as given below.

{
"error": {
"root_cause": [
{
"type": "general_script_exception",
"reason": "failed to compile script [ScriptException[compile error]; nested: IllegalArgumentException[unexpected token ['{'] was expecting one of [{, ';'}].];]"
}
],
"type": "general_script_exception",
"reason": "failed to compile script [ScriptException[compile error]; nested: IllegalArgumentException[unexpected token ['{'] was expecting one of [{, ';'}].];]"
},
"status": 500
}

And i find this forum same issue, but no team to reply it at link CPU usage and Memory usage

Can anyone help me out with the situation.

Thanks!
Dol

#Sorry for a little grammar

2 
PUT _xpack/watcher/watch/mem_watch
    {
      "trigger": {
        "schedule": {
          "interval": "1m"
        }
      },
      "input": {
        "search": {
          "request": {
            "indices": [
              "metricbeat-*"
            ],
            "types" : [
              "node_stats"
            ],
            "body": {
              "size" : 0,
              "query": {
                "bool": {
                  "filter": {
                    "range": {
                      "timestamp": {
                        "gte": "now-2m",
                        "lte": "now"
                      }
                    }
                  }
                }
              },
              "aggs": {
                "minutes": {
                  "date_histogram": {
                    "field": "timestamp",
                    "interval": "minute"
                  },
                  "aggs": {
                    "nodes": {
                      "terms": {
                        "field": "source_node.name",
                        "size": 10,
                        "order": {
                          "memory": "desc"
                        }
                      },
                      "aggs": {
                        "memory": {
                          "avg": {
                            "field": "node_stats.jvm.mem.heap_used_percent"
                          }
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      },
      "throttle_period": "30m", 
      "condition": {
        "script":  "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value >= 75;"
      },
      "actions": {
        "send_email": {
           "transform": {
            "script": "def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.findAll { return it.memory && it.memory.value >= 75 };"
          },
          "email": { 
            "to": "example.email@domain.com", 
            "subject": "Watcher Notification - HIGH MEMORY USAGE",
            "body": "Nodes with HIGH MEMORY Usage (above 75%):\n\n{{#ctx.payload._value}}\"{{key}}\" - Memory Usage is at {{memory.value}}%\n{{/ctx.payload._value}}"
          }
        }
      }
    }

this is my code.

3

The example you are checking out is for watching marvel data. Marvel was used in Elasticsearch 2.x, not in 5.x - and the example is also tailored for 2.x. because of this it used groovy as a scripting language, but you need to change those to use painless.

You can check out the alerting examples in our examples repo for some painless scripting examples..

4

thanks for advice me. @spinscale
i'll try again. if i have issue i will come back to ask you again.

5

PUT _xpack/watcher/watch/mem_warning
{
"trigger": {
"schedule": {
"interval": "10m"
}
},
"input": {
"search": {
"request": {
"indices": [
"metricbeat-*"
],
"types" : [
"metricsets"
],
"body": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-7h",
"lte": "now"
}
}
}
]
}
},
"aggs": {
"minutes": {
"date_histogram": {
"field": "@timestamp",
"interval": "10m"
},
"aggs": {
"hosts": {
"terms": {
"field": "beat.hostname",
"size": 10
},
"aggs": {
"memory_over_75": {
"filters": {
"filters": {
"high": {
"range": {
"system.memory.used.pct": {
"gte": 75
}
}
},
"low": {
"range": {
"system.memory.used.pct": {
"lt": 75
}
}
}
}
}
}
}
}
}
}
}
}
}
}
},
"throttle_period": "30m",
"condition": {
"script": "if (ctx.payload.aggregations.minutes.buckets.size() != 0) return true; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.hosts.buckets[0]; return doc['hosts'] && doc['system.memory.total'].value && doc['system.memory.used.pct'].value >= 75;"
},
"actions": {
"send_email": {
"email": {
"to": "thanadol.thad@wealth.co.th",
"subject": "Watcher Notification - HIGH MEMORY USAGE",
"body": "Nodes with HIGH MEMORY Usage (above 75%):\n\n{{#ctx.payload._value}}"{{key}}" - Memory Usage is at {{memory.value}}%\n{{/ctx.payload._value}}"
}
}
}
}

Hi @spinscale,
i want you to introduce the show output actions body. result host name and mem used values more than 75%

thanks.

6

hey

just to be sure. Is this working as expected now for you? Or was there a question hidden in your last post, which we need to work on? :slight_smile:

--Alex

7

thanks for reply @spinscale
i'm not sure this code is correct. because this result email actions is "
Nodes with HIGH MEMORY Usage (above 75%): "

And result execute_api

"result": {
"execution_time": "2017-07-05T08:18:14.619Z",
"execution_duration": 14044,
"input": {
"type": "search",
"status": "success",
"payload": {
"_shards": {
"total": 50,
"failed": 0,
"successful": 50
},
"hits": {
"hits": ,
"total": 329920,
"max_score": 0
},
"took": 6108,
"timed_out": false,
"aggregations": {
"minutes": {
"buckets": [
{
"key_as_string": "2017-07-05T01:50:00.000Z",
"doc_count": 502,
"hosts": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"doc_count": 502,
"memory_over_75": {
"buckets": {
"high": {
"doc_count": 0
},
"low": {
"doc_count": 7
}
}
},
"key": "DESKTOP-QDUTNR1"
}
]
},
"key": 1499219400000
},

"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"metricbeat-*"
],
"types": [
"metricsets"
],
"body": {
"size": 0,
"query": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-7h",
"lte": "now"
}
}
}
]
}
},
"aggs": {
"minutes": {
"date_histogram": {
"field": "@timestamp",
"interval": "10m"
},
"aggs": {
"hosts": {
"terms": {
"field": "beat.hostname",
"size": 10
},
"aggs": {
"memory_over_75": {
"filters": {
"filters": {
"high": {
"range": {
"system.memory.used.pct": {
"gte": 75
}
}
},
"low": {
"range": {
"system.memory.used.pct": {
"lt": 75
}
}
}
}
}
}
}
}
}
}
}
}
}
}
},
"condition": {
"type": "script",
"status": "success",
"met": true
},
"actions": [
{
"id": "send_email",
"type": "email",
"status": "success",
"email": {
"account": "gmail_account",
"message": {
"id": "mem_warning_2ad2b2fb-4177-401e-b50c-6706a6f46926-2017-07-05T08:18:14.619Z",
"sent_date": "2017-07-05T08:18:22.203Z",
"to": [
"thanadol.thad@wealth.co.th"
],
"subject": "Watcher Notification - HIGH MEMORY USAGE",
"body": {
"text": """
Nodes with HIGH MEMORY Usage (above 75%):

"""
}
}
}
}
]
},
"messages":
}
}

--Dol