Is there some form of functionality built into the ELK stack that can automatically alert me when data is missing? i.e. if we're missing a chunk of time like the image below.
Currently, we're only made aware of missing days when my colleague runs his reports and notices blocks but ideally, we'd like to be notified if the system detects there's a block of time missing from the figures/records.
At the moment, we download the information from AWS in 15 minute internals and store the information to our drive and subsequently populates the data. I've been thinking about putting something together to read the directory (either Python or PowerShell) to scan for a number of files and will send an email alert if it detects a minimum number of files etc for a particular day within a set timeframe.
Ideally, if there's some functionality built into ELK that can do this then that would be superb to know but I would appreciate any input on this as I'm eager to get something put into place.
You could create an index threshold alert in Kibana that would trigger if the number of documents in your index is lower than some specific value for a specific time interval.
If you are running with the basic license the only actions for this alert are to index the alert in some index or log it in the Kibana log file, but you could index the alert in some index and then use a open source tool like elastalert2 to query this index and send an e-mail.
Thank you for your reply, I'm running version 7.17.4 for the stack. Not entirely sure how to go about creating an index in all honesty. The IT guy who handled this side of things left the company so I'm trying to pick it all up with extremely limited knowledge of the stack.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
