# Jvm heap monitoring

**URL:** <https://discuss.elastic.co/t/jvm-heap-monitoring/109251>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [November 27, 2017, 4:46pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251 "2017-11-27T16:46:41Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [November 27, 2017, 4:46pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/1 "2017-11-27T16:46:41Z")

</div>

I am receiving below error message when I am trying to configure watcher for jvm heap size memory.

{  
"error" : {  
"root\_cause" : [  
{  
"type" : "general\_script\_exception",  
"reason" : "failed to compile script [ScriptException[compile error]; nested: IllegalArgumentException[unexpected token ['{'] was expecting one of [{, ';'}].];]"  
}  
],  
"type" : "general\_script\_exception",  
"reason" : "failed to compile script [ScriptException[compile error]; nested: IllegalArgumentException[unexpected token ['{'] was expecting one of [{, ';'}].];]"  
},  
"status" : 500  
}

can someone clarify where the issue could be?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 28, 2017, 7:42am UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/2 "2017-11-28T07:42:18Z")

</div>

Hey,

This is impossible to tell, without seeing the actual watch. The exception tells you, that the script, that you used in your condition could not be compiled, so apparently there is some syntax error in your condition.

If you share it, someone might be able to help.

--Alex

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [November 28, 2017, 2:44pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/3 "2017-11-28T14:44:10Z")

</div>

Thanks for the reply, Alex.  
Yes, i was able to fix it with compilation errors but i am receiving below error

Failed to execute watch [mem\_watch\_eac2355d-3cb8-45c1-b5a7-b5c927624638-2017-11-27T22:30:33.070Z]

assuming my watcher is not working to monitor the jvm heap memory and I am unable to trace where the issue is. Please advise.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 28, 2017, 2:52pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/4 "2017-11-28T14:52:03Z")

</div>

Again, please share your watch and share the output of the execute watch API, plus stack traces if you got them.

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [November 28, 2017, 8:02pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/5 "2017-11-28T20:02:45Z")

</div>

Sorry that i am quite new which leads to missing something to share more information.  
i executed the watch api as below.

POST \_xpack/watcher/watch/mem\_watch/\_execute

getting below error... with warning  
"#! Deprecation: [groovy] scripts are deprecated, use [painless] scripts instead"

is this warning something gives this exception. please suggest.  
},  
"exception": {  
"type": "script\_exception",  
"reason": "error evaluating if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value \>= 75;",  
"script\_stack": [],  
"script": "",  
"lang": "groovy",  
"caused\_by": {  
"type": "null\_pointer\_exception",  
"reason": "Cannot get property 'minutes' on null object",  
"stack\_trace": """  
java.lang.NullPointerException: Cannot get property 'minutes' on null object  
at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:60)  
at d76474fa78ba7805fc9568d7f3bc44ec409afce0.run(d76474fa78ba7805fc9568d7f3bc44ec409afce0:1)  
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$$Lambda$1951/1806137276.run(Unknown Source)  
at java.security.AccessController.doPrivileged(Native Method)  
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:304)  
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:94)  
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:84)  
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:391)  
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275)  
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:136)  
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:64)  
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87)  
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:745)

"""  
},  
"stack\_trace": """  
ScriptException[error evaluating if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value \>= 75;]; nested: NullPointerException[Cannot get property 'minutes' on null object];  
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:322)  
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:94)  
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:84)  
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:391)  
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275)  
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:136)  
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:64)  
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87)  
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166)  
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)  
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)  
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)  
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)  
at java.lang.Thread.run(Thread.java:745)  
Caused by: java.lang.NullPointerException: Cannot get property 'minutes' on null object  
at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:60)  
at d76474fa78ba7805fc9568d7f3bc44ec409afce0.run(d76474fa78ba7805fc9568d7f3bc44ec409afce0:1)  
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$$Lambda$1951/1806137276.run(Unknown Source)  
at java.security.AccessController.doPrivileged(Native Method)  
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:304)  
... 13 more

"""

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [November 28, 2017, 8:44pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/6 "2017-11-28T20:44:40Z")

</div>

Alex,  
is it something marvel-agent is needed in order to resolve this issue with ES 5.3.3 version?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 29, 2017, 9:55am UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/7 "2017-11-29T09:55:20Z")

</div>

you still have not shared the watch or an history entry, I'll give up asking for now 🙂

> [@msjyosh](#):
>
> java.lang.NullPointerException: Cannot get property 'minutes' on null object

This is the culprit. You are trying to access an element that does not exist. `ctx.payload.aggregations` does not exist, so you cannot access the `minutes` element in there. It seems your search does not contain an aggregation.

You might have guessed it already. If you had shared the output of the execute watch api plus the original watch, there might be a chance to help. This way I can only explain what is broken without being able to delve into deeper detail.

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [November 29, 2017, 3:25pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/8 "2017-11-29T15:25:11Z")

</div>

First - thanks for your valuable reply and giving you the watch API execution with error

# Watch API Execution:

POST \_xpack/watcher/watch/mem\_watch/\_execute  
#! Deprecation: [groovy] scripts are deprecated, use [painless] scripts instead  
{  
"\_id": "mem\_watch\_f0c6931a-6cb5-4d6c-a1f8-8c9b764478eb-2017-11-28T23:05:24.878Z",  
"watch\_record": {  
"watch\_id": "mem\_watch",  
"state": "failed",  
"trigger\_event": {  
"type": "manual",  
"triggered\_time": "2017-11-28T23:05:24.878Z",  
"manual": {  
"schedule": {  
"scheduled\_time": "2017-11-28T23:05:24.878Z"  
}  
}  
},  
"input": {  
"search": {  
"request": {  
"search\_type": "query\_then\_fetch",  
"indices": [  
".marvel-es-1-_"  
],  
"types": [  
"node\_stats"  
],  
"body": {  
"size": 0,  
"query": {  
"bool": {  
"filter": {  
"range": {  
"timestamp": {  
"gte": "now-2m",  
"lte": "now"  
}  
}  
}  
}  
},  
"aggs": {  
"minutes": {  
"date\_histogram": {  
"field": "timestamp",  
"interval": "minute"  
},  
"aggs": {  
"nodes": {  
"terms": {  
"field": "source\_node.name",  
"size": 10,  
"order": {  
"memory": "desc"  
}  
},  
"aggs": {  
"memory": {  
"avg": {  
"field": "node\_stats.jvm.mem.heap\_used\_percent"  
}  
}  
}  
}  
}  
}  
}  
}  
}  
}  
},  
"condition": {  
"script": {  
"inline": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value \>= 75;",  
"lang": "groovy"  
}  
},  
"result": {  
"execution\_time": "2017-11-28T23:05:24.878Z",  
"execution\_duration": 1,  
"input": {  
"type": "search",  
"status": "success",  
"payload": {  
"\_shards": {  
"total": 0,  
"failed": 0,  
"successful": 0  
},  
"hits": {  
"hits": [],  
"total": 0,  
"max\_score": 0  
},  
"took": 1,  
"timed\_out": false  
},  
"search": {  
"request": {  
"search\_type": "query\_then\_fetch",  
"indices": [  
".marvel-es-1-_"  
],  
"types": [  
"node\_stats"  
],  
"body": {  
"size": 0,  
"query": {  
"bool": {  
"filter": {  
"range": {  
"timestamp": {  
"gte": "now-2m",  
"lte": "now"  
}  
}  
}  
}  
},  
"aggs": {  
"minutes": {  
"date\_histogram": {  
"field": "timestamp",  
"interval": "minute"  
},  
"aggs": {  
"nodes": {  
"terms": {  
"field": "source\_node.name",  
"size": 10,  
"order": {  
"memory": "desc"  
}  
},  
"aggs": {  
"memory": {  
"avg": {  
"field": "node\_stats.jvm.mem.heap\_used\_percent"  
}  
}  
}  
}  
}  
}  
}  
}  
}  
}  
},  
"actions": []  
},  
"exception": {  
"type": "script\_exception",  
"reason": "error evaluating if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value \>= 75;",  
"script\_stack": [],  
"script": "",  
"lang": "groovy",  
"caused\_by": {  
"type": "null\_pointer\_exception",  
"reason": "Cannot get property 'minutes' on null object",  
"stack\_trace": """  
java.lang.NullPointerException: Cannot get property 'minutes' on null object  
at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:60)  
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:228)  
at d76474fa78ba7805fc9568d7f3bc44ec409afce0.run(d76474fa78ba7805fc9568d7f3bc44ec409afce0:1)  
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$$Lambda$1951/1806137276.run(Unknown Source)  
at java.security.AccessController.doPrivileged(Native Method)  
at

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [November 29, 2017, 3:26pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/9 "2017-11-29T15:26:41Z")

</div>

# Original Watch I created as below if you are referring this.

curl -XPUT 'localhost:9206/\_watcher/watch/mem\_watch?pretty' -H 'Content-Type: application/json' -d'  
{  
"trigger": {  
"schedule": {  
"interval": "1m"  
}  
},  
"input": {  
"search": {  
"request": {  
"indices": [  
".marvel-es-1-\*"  
],  
"types": [  
"node\_stats"  
],  
"body": {  
"size": 0,  
"query": {  
"bool": {  
"filter": {  
"range": {  
"timestamp": {  
"gte": "now-2m",  
"lte": "now"  
}  
}  
}  
}  
},  
"aggs": {  
"minutes": {  
"date\_histogram": {  
"field": "timestamp",  
"interval": "minute"  
},  
"aggs": {  
"nodes": {  
"terms": {  
"field": "source\_node.name",  
"size": 10,  
"order": {  
"memory": "desc"  
}  
},  
"aggs": {  
"memory": {  
"avg": {  
"field": "node\_stats.jvm.mem.heap\_used\_percent"  
}  
}  
}  
}  
}  
}  
}  
}  
}  
}  
},  
"throttle\_period": "30m",  
"condition": {  
"script": {  
"lang": "groovy",  
"inline": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value \>= 75;"  
}  
},  
"actions": {  
"send\_email": {  
"transform": {  
"script": {  
"lang": "groovy",  
"inline": "def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.findAll { return it.memory && it.memory.value \>= 75 };"  
}  
},  
"email": {  
"to": "test@test.com",  
"subject": "Watcher Notification - HIGH MEMORY USAGE",  
"body": "Nodes with HIGH MEMORY Usage (above 75%):{{#ctx.payload.\_value}} {{key}} - Memory Usage is at {{memory.value}}%{{ctx.payload.\_value}}"  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [November 29, 2017, 3:33pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/10 "2017-11-29T15:33:41Z")

</div>

I am sorry if i am still missing something to share due to not understanding what i have to provide ☹  
Please tell me if anything that i can provide to fix this issue. 🙂

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [November 30, 2017, 8:23am UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/11 "2017-11-30T08:23:23Z")

</div>

please take your time and format your messages properly or use a gist, this is nearly impossible to read without indendation (humans are not good with JSON 🙂

The aggregation in your watch is named `memory`, but you are using a different name in the condition.

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [December 1, 2017, 10:13pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/15 "2017-12-01T22:13:49Z")

</div>

Hi Alex,  
Though i update the converted JSON data it is not being updated properly.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 4, 2017, 8:28am UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/16 "2017-12-04T08:28:01Z")

</div>

Please see [here](https://www.elastic.co/help) for more information of proper pasting.

You search does not return any hits, thus the required data structure is not in the response JSON.

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [December 4, 2017, 10:42pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/17 "2017-12-04T22:42:53Z")

</div>

[https://gist.github.com/msjyosh/b78c292289b589513497e07e9e19afa3#file-gistfile1-txt](https://gist.github.com/msjyosh/b78c292289b589513497e07e9e19afa3#file-gistfile1-txt)

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [December 4, 2017, 10:43pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/18 "2017-12-04T22:43:19Z")

</div>

Please let me know if this is something would help to resolve my issue 🙂

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 4, 2017, 11:05pm UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/19 "2017-12-04T23:05:56Z")

</div>

have you read the second part of my last response? If your query does not return anything, the condition in the watch will not trigger and this was the case with the output you pasted earlier. There are simply no hits with your query.

---

<div class="post-metadata">

**Author:** ![msjyosh](https://avatars.discourse-cdn.com/v4/letter/m/65b543/32.png) [@msjyosh](https://discuss.elastic.co/u/msjyosh)\
**Post date:** [December 5, 2017, 12:44am UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/20 "2017-12-05T00:44:26Z")

</div>

oh yes i read it. i thought something wrong with script which is why it is not returning any result.  
Please advise

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [December 5, 2017, 7:44am UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/21 "2017-12-05T07:44:45Z")

</div>

your problem has nothing to do with watcher. It is not even clear if this is a problem, because maybe there are no hits that match. If you think there should be matches, you need to change the query first, and you should probably do that outside of watcher by just using a regular search request until the query returns what you need.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 2, 2018, 7:45am UTC](https://discuss.elastic.co/t/jvm-heap-monitoring/109251/22 "2018-01-02T07:45:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.