Alerts in elastic

[Karthik2411]

I am working creating alerts in elastic. I have a complex requirement of checking failures every hour and then summing them up to alert when the sum is exceeding 10 and then checking every hour and doing the same. I am also trying to delete the history after it reports the first alert.

This is the code I could develop for that

{
"trigger": {
"schedule": {
"cron": "0 0 0-23 * * ?"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"crs-*"
],
"types": ,
"body": {
"size": 0,
"query": {
"bool": {
"must": [
{
"term": {
"loadMessage.keyword": "app_script_load_fail"
}
},
{
"term": {
"logType": {
"value": "mashup_log_from_atlas"
}
}
},
{
"range": {
"@timestamp": {
"format": "epoch_millis",
"gte": "now-60m/m",
"lte": "now/m"
}
}
}
],
"must_not": [
{
"terms": {
"report.keyword": [
"basic_webex",
"premium_webex",
"partner_webex"
]
}
}
]
}
},
"aggs": {
"by_report": {
"terms": {
"field": "report.keyword",
"size": 100
}
}
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gte": 10
}
}
},
"actions": {
"heartbeat": {
"email": {
"profile": "standard",
"to": [
"srinekk@cisco.com"
]
],
"subject": "Failures from qlik",
"body": {
"text": "More than 10 failures found in last 1 hour. \n Failures reported from :{{#ctx.payload.aggregations.by_report.buckets}}{{key}}-{{doc_count}},{{/ctx.payload.aggregations.by_report.buckets}}"
}
}
}
}
}

Please throw some light on what could be the better approach for this

[spinscale (Alexander Reelsen)]

Hey,

first, please take the time to properly format your code snippets using markdown, this is pretty much unreadable for humans (which are not good JSON parsers in the first place).

Can you explain why your current approach is not sufficient and what you would like to improve, as noone knows your use-case better than you, every hint would just be some guesswork.

--Alex

[Karthik2411]

Schedule a check for the number of failures every hour and then generate an alert if the sum of the failures of the hours checked is more than 10. Refresh the check after every alert and follow the same again until you get to generate another alert. These alerts should be generated from (7 am - 5pm) PST.

Hour	FailureCount
7-8	4
8-9	2
9-10	4
Total	10

It should alert when the sum is 10 or more and then start checking freshly from next hour(10-11) and do the same.
So, watcher should trigger every hour but will send notification only if above condition satisfies.

[spinscale (Alexander Reelsen)]

I do no understand why you need to divide by hour, if you are using a sum to decide whether you should trigger an alert.

Also, can you be more exact what means refresh in your context? If you mean to ignore some previous results, than just execute a query that allows you to inspect the results from the previous run and factor this into your decision (the condition) to decide if the action should be triggered.

This way you could prevent the repeated execution that you want do not want to do it seems.

Another way would be to check the watcher history for previous runs, however I think checking your previous data is easier from my current point of view.

[Karthik2411]

@spinscale. Okay let us say we are not dividing by hour. But what would be the approach if we are trying to add the errors and then alert after the sum is 10. How far should we go back is the issue I am facing. I know that I am asking you too many questions. But the ultimate goal is to report if the number of errors are more than 10 and after that it should restart counting by excepting the previous 10.

[Karthik2411]

@spinscale how can I access previous run results? Can you guide me to some example?

Can you please reply ??