No data Alert

[Aniket_Pant (kaal)]

Hi Team,

We want to create alert if no data is receiving is from last 15 minute. We have tried with the watcher it worked for us but we can't go as in watcher the alert status does not change like as in alert we have active , recover options we have.

I tried with Index Threshold alert but i can't able to create it. Please if any one has solution update me here

[jsanz (Jorge Sanz)]

This setup worked for me. I could get my alert triggered and recovered as my metricbeat was sending data or not to my cluster.

And on the history of the alert it shows when it was triggered and recovered

What was not working for you?

[Aniket_Pant (kaal)]

We have deployed metricbeat in almost 300 servers and if somehow metricbeat is stop in one server then we should get alert " metricbeat is stopped in xx.xx.xx.xx"

[jsanz (Jorge Sanz)]

There is a grouping capability on the threshold index

Over/Grouped Over

Specify whether the aggregation is applied over all documents or split into groups using a grouping field. If grouping is used, an alert will be created for each group when it exceeds the threshold. To limit the number of alerts on high cardinality fields, you must specify the number of groups to check against the threshold. Only the top groups are checked.

Index threshold | Kibana Guide [8.8] | Elastic

On my example:

I haven't an environment like yours to test so I'm talking from a very limited experience.

[Aniket_Pant (kaal)]

Thanks @jsanz let me try this because in kibana Alert i don't we can customer message as i mention we can show metricbeat is stopped in xx.xx.xx.xx. it should automatically take host.hostname value from the document.

[stephenb (Stephen Brown)]

What Version of the Stack?

Metric threshold have missing data alerts built in....Recent versions which report when the data is missing

[Aniket_Pant (kaal)]

7.17

[Aniket_Pant (kaal)]

Hi Team,

We are now getting the alert but we need to modify the message body.We are getting this alert on mail

In mail we are getting alert like this

Alert: No Hits observed in xx.xx.xx.xx in the last 10m IP_Address: xx.xx.xx.xx State: ALERT

--

This message was sent by Kibana. View rule in Kibana.

But in kibana email connector we have set like this

Alert: No Hits observed in {{context.group}} in the last 10m
IP_Address: {{context.group}}
State: {{context.alertState}}

We are getting messages on email as a single line , it is not coming as multiline

[stephenb (Stephen Brown)]

@Aniket_Pant

Alert: No Hits observed in {{context.group}} in the last 10m
IP_Address: {{context.group}}
State: {{context.alertState}}

Perhaps try another line in between

Alert: No Hits observed in {{context.group}} in the last 10m

IP_Address: {{context.group}}

State: {{context.alertState}}

[Aniket_Pant (kaal)]

Thanks @stephenb ,
Could you please tell me whether this possible or not

Can we get alert for status change like that particular node who was not sending the logs , is now sending the logs

[stephenb (Stephen Brown)]

Need more info.

What Version are you on?

Can you show the complete setup of the alert?

as @jsanz explained above you can be alerted when data stops and "recovered" when it continues... did you try / test his suggestion? The group by field is what you will use to look by node... you just need to group by that field that represents your node name

[Aniket_Pant (kaal)]

Hi @stephenb ,

We are using 7.17.5 ELK cluster and it is working i mean we are getting the alert but when a alert is recovered how can we get alert for this also in kibana UI we can see it is now fine application is sending the logs but we need a alert also if status is changed to recover

I will share you the alert info

[stephenb (Stephen Brown)]

You need to set up another action for Recoverd ... that is the way you do it
1 Action for Threshold Met
1 Acgtion for Recovered

[Aniket_Pant (kaal)]

Hi @stephenb ,
yesterday we got alert you can see below

Alert: No Hits observed in xx.xx.xx.xx in the last 10m
IP_Address: xx.xx.xx.xx
State: ALERT
--
This message was sent by Kibana. View rule in Kibana.

and when it was recovered , in the alert recovered message we didn't get any server ip and alert state

Alert: (Clear/Reset) Hits now observed in the last 10m
IP_Address:
State:
--
This message was sent by Kibana. View rule in Kibana.

We are need to get alert when state is Alert and Recovered do i need to have seperate condition for recovered ?

[stephenb (Stephen Brown)]

1 condition with 2 Actions Alert and Recovered

It is hard to tell your set up as you don't show the complete screens

What version are you on? The recovered data was only added in fairly recent versions.

[Aniket_Pant (kaal)]

Hi @stephenb ,

What version are you on? The recovered data was only added in fairly recent versions.

7.17.5

I have attached Screenshots you can check is there something i am doing wrong
Screenshot-1

Screenshot-2

Screenshot-3

Screenshot-4

[stephenb (Stephen Brown)]

Hi @Aniket_Pant

Unfortunately, the feature context for recovered alerts did not come until 8.5 I believe so that is probably why it is missing in 7.17

You can always test the full message by just putting {{context}} into the Message to see what is all there, I think you will find it blank in 7.17 and fully populated from 8.5 on

[Aniket_Pant (kaal)]

Thank you @stephenb and @jsanz for your support we will plan to upgrade

[Aniket_Pant (kaal)]

we need to upgrade to 8.5 correct ?