Alert when data is missing?

Hi all,

Is there some form of functionality built into the ELK stack that can automatically alert me when data is missing? i.e. if we're missing a chunk of time like the image below.

Currently, we're only made aware of missing days when my colleague runs his reports and notices blocks but ideally, we'd like to be notified if the system detects there's a block of time missing from the figures/records.

At the moment, we download the information from AWS in 15 minute internals and store the information to our drive and subsequently populates the data. I've been thinking about putting something together to read the directory (either Python or PowerShell) to scan for a number of files and will send an email alert if it detects a minimum number of files etc for a particular day within a set timeframe.

Ideally, if there's some functionality built into ELK that can do this then that would be superb to know but I would appreciate any input on this as I'm eager to get something put into place.

Which version of the stack are you in?

You could create an index threshold alert in Kibana that would trigger if the number of documents in your index is lower than some specific value for a specific time interval.

If you are running with the basic license the only actions for this alert are to index the alert in some index or log it in the Kibana log file, but you could index the alert in some index and then use a open source tool like elastalert2 to query this index and send an e-mail.

Thank you for your reply, I'm running version 7.17.4 for the stack. Not entirely sure how to go about creating an index in all honesty. The IT guy who handled this side of things left the company so I'm trying to pick it all up with extremely limited knowledge of the stack.

You can check the documentation on how to create an alert.

Start here, then this shows you how to create an index threshold rule which can help you monitor your index.

1 Like