Jvm heap monitoring

Elastic Stack Elasticsearch
1

I am receiving below error message when I am trying to configure watcher for jvm heap size memory.

{
"error" : {
"root_cause" : [
{
"type" : "general_script_exception",
"reason" : "failed to compile script [ScriptException[compile error]; nested: IllegalArgumentException[unexpected token ['{'] was expecting one of [{, ';'}].];]"
}
],
"type" : "general_script_exception",
"reason" : "failed to compile script [ScriptException[compile error]; nested: IllegalArgumentException[unexpected token ['{'] was expecting one of [{, ';'}].];]"
},
"status" : 500
}

can someone clarify where the issue could be?

2

Hey,

This is impossible to tell, without seeing the actual watch. The exception tells you, that the script, that you used in your condition could not be compiled, so apparently there is some syntax error in your condition.

If you share it, someone might be able to help.

--Alex

3

Thanks for the reply, Alex.
Yes, i was able to fix it with compilation errors but i am receiving below error

Failed to execute watch [mem_watch_eac2355d-3cb8-45c1-b5a7-b5c927624638-2017-11-27T22:30:33.070Z]

assuming my watcher is not working to monitor the jvm heap memory and I am unable to trace where the issue is. Please advise.

4

Again, please share your watch and share the output of the execute watch API, plus stack traces if you got them.

5

Sorry that i am quite new which leads to missing something to share more information.
i executed the watch api as below.

POST _xpack/watcher/watch/mem_watch/_execute

getting below error... with warning
"#! Deprecation: [groovy] scripts are deprecated, use [painless] scripts instead"

is this warning something gives this exception. please suggest.
},
"exception": {
"type": "script_exception",
"reason": "error evaluating if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value >= 75;",
"script_stack": [],
"script": "",
"lang": "groovy",
"caused_by": {
"type": "null_pointer_exception",
"reason": "Cannot get property 'minutes' on null object",
"stack_trace": """
java.lang.NullPointerException: Cannot get property 'minutes' on null object
at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:60)
at d76474fa78ba7805fc9568d7f3bc44ec409afce0.run(d76474fa78ba7805fc9568d7f3bc44ec409afce0:1)
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$$Lambda$1951/1806137276.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:304)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:94)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:84)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:391)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275)
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:136)
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:64)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

"""
},
"stack_trace": """
ScriptException[error evaluating if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value >= 75;]; nested: NullPointerException[Cannot get property 'minutes' on null object];
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:322)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.doExecute(ScriptCondition.java:94)
at org.elasticsearch.xpack.watcher.condition.ScriptCondition.execute(ScriptCondition.java:84)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.executeInner(ExecutionService.java:391)
at org.elasticsearch.xpack.watcher.execution.ExecutionService.execute(ExecutionService.java:275)
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:136)
at org.elasticsearch.xpack.watcher.transport.actions.execute.TransportExecuteWatchAction.masterOperation(TransportExecuteWatchAction.java:64)
at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:87)
at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.doRun(TransportMasterNodeAction.java:166)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:638)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException: Cannot get property 'minutes' on null object
at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:60)
at d76474fa78ba7805fc9568d7f3bc44ec409afce0.run(d76474fa78ba7805fc9568d7f3bc44ec409afce0:1)
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$$Lambda$1951/1806137276.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:304)
... 13 more

"""

6

Alex,
is it something marvel-agent is needed in order to resolve this issue with ES 5.3.3 version?

7

you still have not shared the watch or an history entry, I'll give up asking for now :slight_smile:

This is the culprit. You are trying to access an element that does not exist. ctx.payload.aggregations does not exist, so you cannot access the minutes element in there. It seems your search does not contain an aggregation.

You might have guessed it already. If you had shared the output of the execute watch api plus the original watch, there might be a chance to help. This way I can only explain what is broken without being able to delve into deeper detail.

8

First - thanks for your valuable reply and giving you the watch API execution with error

Watch API Execution:

POST _xpack/watcher/watch/mem_watch/_execute
#! Deprecation: [groovy] scripts are deprecated, use [painless] scripts instead
{
"_id": "mem_watch_f0c6931a-6cb5-4d6c-a1f8-8c9b764478eb-2017-11-28T23:05:24.878Z",
"watch_record": {
"watch_id": "mem_watch",
"state": "failed",
"trigger_event": {
"type": "manual",
"triggered_time": "2017-11-28T23:05:24.878Z",
"manual": {
"schedule": {
"scheduled_time": "2017-11-28T23:05:24.878Z"
}
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
".marvel-es-1-"
],
"types": [
"node_stats"
],
"body": {
"size": 0,
"query": {
"bool": {
"filter": {
"range": {
"timestamp": {
"gte": "now-2m",
"lte": "now"
}
}
}
}
},
"aggs": {
"minutes": {
"date_histogram": {
"field": "timestamp",
"interval": "minute"
},
"aggs": {
"nodes": {
"terms": {
"field": "source_node.name",
"size": 10,
"order": {
"memory": "desc"
}
},
"aggs": {
"memory": {
"avg": {
"field": "node_stats.jvm.mem.heap_used_percent"
}
}
}
}
}
}
}
}
}
}
},
"condition": {
"script": {
"inline": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value >= 75;",
"lang": "groovy"
}
},
"result": {
"execution_time": "2017-11-28T23:05:24.878Z",
"execution_duration": 1,
"input": {
"type": "search",
"status": "success",
"payload": {
"_shards": {
"total": 0,
"failed": 0,
"successful": 0
},
"hits": {
"hits": [],
"total": 0,
"max_score": 0
},
"took": 1,
"timed_out": false
},
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
".marvel-es-1-"
],
"types": [
"node_stats"
],
"body": {
"size": 0,
"query": {
"bool": {
"filter": {
"range": {
"timestamp": {
"gte": "now-2m",
"lte": "now"
}
}
}
}
},
"aggs": {
"minutes": {
"date_histogram": {
"field": "timestamp",
"interval": "minute"
},
"aggs": {
"nodes": {
"terms": {
"field": "source_node.name",
"size": 10,
"order": {
"memory": "desc"
}
},
"aggs": {
"memory": {
"avg": {
"field": "node_stats.jvm.mem.heap_used_percent"
}
}
}
}
}
}
}
}
}
}
},
"actions": []
},
"exception": {
"type": "script_exception",
"reason": "error evaluating if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value >= 75;",
"script_stack": [],
"script": "",
"lang": "groovy",
"caused_by": {
"type": "null_pointer_exception",
"reason": "Cannot get property 'minutes' on null object",
"stack_trace": """
java.lang.NullPointerException: Cannot get property 'minutes' on null object
at org.codehaus.groovy.runtime.NullObject.getProperty(NullObject.java:60)
at org.codehaus.groovy.vmplugin.v7.IndyInterface.selectMethod(IndyInterface.java:228)
at d76474fa78ba7805fc9568d7f3bc44ec409afce0.run(d76474fa78ba7805fc9568d7f3bc44ec409afce0:1)
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript$$Lambda$1951/1806137276.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at

9

Original Watch I created as below if you are referring this.

curl -XPUT 'localhost:9206/_watcher/watch/mem_watch?pretty' -H 'Content-Type: application/json' -d'
{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"search": {
"request": {
"indices": [
".marvel-es-1-*"
],
"types": [
"node_stats"
],
"body": {
"size": 0,
"query": {
"bool": {
"filter": {
"range": {
"timestamp": {
"gte": "now-2m",
"lte": "now"
}
}
}
}
},
"aggs": {
"minutes": {
"date_histogram": {
"field": "timestamp",
"interval": "minute"
},
"aggs": {
"nodes": {
"terms": {
"field": "source_node.name",
"size": 10,
"order": {
"memory": "desc"
}
},
"aggs": {
"memory": {
"avg": {
"field": "node_stats.jvm.mem.heap_used_percent"
}
}
}
}
}
}
}
}
}
}
},
"throttle_period": "30m",
"condition": {
"script": {
"lang": "groovy",
"inline": "if (ctx.payload.aggregations.minutes.buckets.size() == 0) return false; def latest = ctx.payload.aggregations.minutes.buckets[-1]; def node = latest.nodes.buckets[0]; return node && node.memory && node.memory.value >= 75;"
}
},
"actions": {
"send_email": {
"transform": {
"script": {
"lang": "groovy",
"inline": "def latest = ctx.payload.aggregations.minutes.buckets[-1]; return latest.nodes.buckets.findAll { return it.memory && it.memory.value >= 75 };"
}
},
"email": {
"to": "test@test.com",
"subject": "Watcher Notification - HIGH MEMORY USAGE",
"body": "Nodes with HIGH MEMORY Usage (above 75%):{{#ctx.payload._value}} {{key}} - Memory Usage is at {{memory.value}}%{{ctx.payload._value}}"
}
}
}
}

10

I am sorry if i am still missing something to share due to not understanding what i have to provide :frowning:
Please tell me if anything that i can provide to fix this issue. :slight_smile:

11

please take your time and format your messages properly or use a gist, this is nearly impossible to read without indendation (humans are not good with JSON :slight_smile:

The aggregation in your watch is named memory, but you are using a different name in the condition.

15

Hi Alex,
Though i update the converted JSON data it is not being updated properly.

16

Please see here for more information of proper pasting.

You search does not return any hits, thus the required data structure is not in the response JSON.

17

https://gist.github.com/msjyosh/b78c292289b589513497e07e9e19afa3#file-gistfile1-txt

18

Please let me know if this is something would help to resolve my issue :slight_smile:

19

have you read the second part of my last response? If your query does not return anything, the condition in the watch will not trigger and this was the case with the output you pasted earlier. There are simply no hits with your query.

20

oh yes i read it. i thought something wrong with script which is why it is not returning any result.
Please advise

21

your problem has nothing to do with watcher. It is not even clear if this is a problem, because maybe there are no hits that match. If you think there should be matches, you need to change the query first, and you should probably do that outside of watcher by just using a regular search request until the query returns what you need.

22

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.