Email not trigger

Elastic Stack Elasticsearch
1

Hi Team,

I create watcher in Kibana and its getting trigger which i can see under Management->
Elasticsearch-> Watcher-> Watches-> system_process_watch. but email notification doesn't receive.

Metric beat input
*
{
"_index": "metricbeat-6.2.2-2018.04.11",
"_type": "doc",
"_id": "qFphs2IBYXi8yJ1I90wA",
"_version": 1,
"_score": null,
"_source": {
"@timestamp": "2018-04-11T06:26:03.738Z",
"metricset": {
"rtt": 57105,
"name": "process",
"module": "system"
},
"system": {
"process": {
"name": "java",
"pgid": 28531,
"cpu": {
"total": {
"pct": 0.085,
"norm": {
"pct": 0.0212
},
"value": 112400
},
"start_time": "2018-04-11T06:04:43.000Z"
},
"cwd": "////kibana/elasticsearch-6.2.2",
"fd": {
"open": 309,
"limit": {
"soft": 1048576,
"hard": 1048576
}
},
"pid": 28531,
"username": "",
"memory": {
"size": 5027340288,
"rss": {
"pct": 0.0457,
"bytes": 1532678144
},
"share": 23744512
},
"ppid": 25032,
"state": "sleeping",
"cmdline": "///jdk8/jdk1.8.0_101/bin/java -Xms1g -Xmx1g -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.io.tmpdir=/tmp/elasticsearch.TRiKGaHE -XX:+HeapDumpOnOutOfMemoryError -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationStoppedTime -Xloggc:logs/gc.log -XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=32 -XX:GCLogFileSize=64m -Des.path.=///kibana/elasticsearch-6.2.2 -Des.path.conf=///kibana/elasticsearch-6.2.2/config -cp ///kibana/elasticsearch-6.2.2/lib/* org.elasticsearch.bootstrap.Elasticsearch"
}
},
"beat": {
"name": "",
"hostname": "***",
"version": "6.2.2"
}
},
"fields": {
"@timestamp": [
"2018-04-11T06:26:03.738Z"
],
"system.process.cpu.start_time": [
"2018-04-11T06:04:43.000Z"
]
},
"sort": [
1523427963738
]
}
*

Watcher Query
*
PUT _xpack/watcher/watch/system_process_watch
{
"trigger" : {
"schedule" : { "interval" : "2m" }
},
"input" : {
"search" : {
"request" : {
"indices" : "metricbeat-",
"body" : {
"size" : 0,
"query" : { "match" : { "metricset.name" : "process" } }
}
}
}
},
"condition" : {
"compare" : { "system.process.cpu.total.norm.pct" : { "gt" : 0 } }
},
"actions" : {
"email_administrator" : {
"email" : {
"to" : "nalini.ranjan@*****.com",
"subject" : "Encountered {{system.process.cpu.total.norm.pct}} errors",
"body" : "Too many error in the system, see attached data",
"attachments" : {
"attached_data" : {
"data" : {
"format" : "json"
}
}
},
"priority" : "high"
}
}
}
}
*

Watcher execution output
*
{
"watch_id": "system_process_watch",
"node": "TEfIfaz8R16BbkeH9ah1TA",
"state": "execution_not_needed",
"status": {
"state": {
"active": true,
"timestamp": "2018-04-10T07:28:36.198Z"
},
"last_checked": "2018-04-11T06:34:58.376Z",
"actions": {
"email_administrator": {
"ack": {
"timestamp": "2018-04-10T07:28:36.198Z",
"state": "awaits_successful_execution"
}
}
},
"execution_state": "execution_not_needed",
"version": -1
},
"trigger_event": {
"type": "schedule",
"triggered_time": "2018-04-11T06:34:58.376Z",
"schedule": {
"scheduled_time": "2018-04-11T06:34:58.338Z"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"metricbeat-"
],
"types": [],
"body": {
"size": 0,
"query": {
"match": {
"metricset.name": "process"
}
}
}
}
}
},
"condition": {
"compare": {
"system.process.cpu.total.norm.pct": {
"gt": 0
}
}
},
"result": {
"execution_time": "2018-04-11T06:34:58.376Z",
"execution_duration": 2,
"input": {
"type": "search",
"status": "success",
"payload": {
"_shards": {
"total": 3,
"failed": 0,
"successful": 3,
"skipped": 0
},
"hits": {
"hits": [],
"total": 26477,
"max_score": 0
},
"took": 1,
"timed_out": false
},
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"metricbeat-"
],
"types": [],
"body": {
"size": 0,
"query": {
"match": {
"metricset.name": "process"
}
}
}
}
}
},
"condition": {
"type": "compare",
"status": "success",
"met": false,
"compare": {
"resolved_values": {
"system.process.cpu.total.norm.pct": null
}
}
},
"actions": []
},
"messages": []
}
*

Could you please help me to find out my mistake ?

2

please take the time to properly format your code snippets. this makes it nearly impossible to read.

Your condition tries to access a non-existing variable. You need to access the data in the payload and as you are executing a search, you need to access the hits array from a search response. You should take a look at the structure of the returned JSON first, by running the query outside of a watch.

In order to better debug watches, please read this blog post as it helps in reducing your feedback loop https://www.elastic.co/blog/watching-the-watches-writing-debugging-and-testing-watches

3

@spinscale i have gone through the above link but still i face same issue.

i run below execute query from KIBANA Dev tools

GET /_search
{
"query": {
"match": {
"metricset.name" : "memory"
}
}
}

Output for above query
{
"took": 3,
"timed_out": false,
"_shards": {
"total": 23,
"successful": 23,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 398,
"max_score": 2.7516687,
"hits": [
{
"_index": "metricbeat-6.2.2-2018.04.12",
"_type": "doc",
"_id": "VWxXuGIBs2lvJLUHPjTr",
"_score": 2.7516687,
"_source": {
"@timestamp": "2018-04-12T05:32:27.265Z",
"beat": {
"name": "00000000",
"hostname": "00000000",
"version": "6.2.2"
},
"metricset": {
"name": "memory",
"module": "system",
"rtt": 187
},
"system": {
"memory": {
"total": 33569062912,
"used": {
"bytes": 18366738432,
"pct": 0.5471
},
"free": 15202324480,
"actual": {
"used": {
"pct": 0.2068,
"bytes": 6942912512
},
"free": 26626150400
},
"swap": {
"used": {
"bytes": 0,
"pct": 0
},
"free": 8589930496,
"total": 8589930496
}
}
}
}
}

Just to validate wanted to add alert which memory used percentage is more than 0.

**Query for watcher **
PUT _xpack/watcher/watch/_execute
{
"watch" : {
"trigger" : {
"schedule" : {
"interval": "2m"
}
},
"input" : {
"search" : {
"request" : {
"indices" : "metricbeat-*",
"body" : {
"size" : 0,
"query" : { "match" : { "metricset.name" : "memory" } }
}
}
}
},
"condition" : {
"compare" : { "system.memory.used.pct" : { "gt" : 0 } }
},
"actions" : {
"email_administrator" : {
"email" : {
"to" : "++++++++++++",
"subject" : "Encountered {{system.memory.used.pct} errors",
"body" : "Too many error in the system, see attached data",
"attachments" : {
"attached_data" : {
"data" : {
"format" : "json"
}
}
},
"priority" : "high"
}
}
}
}
}
}

**Result for condition **
"condition": {
"type": "compare",
"status": "success",
"met": false,
"compare": {
"resolved_values": {
"system.memory.used.pct": null
}
}
},
Could you please help me where i am doing wrong ?

4

Your problem still remains the same, you are trying to access a non existing field in the condition.

You have more than one search hit, so I am not sure, what you are trying to achieve with this condition, this also makes it impossible for me, how to correct this. Some more context is needed.

Apart from that checking out our examples might help you a lot in order to understand what fields you can access!

5

@spinscale thank you so much for your help..it really help me a lot after going through the example.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.