I'm currently trying to configure an advanced job that detects low counts or zero counts of documents by using low_count by field1. I basically want the job to detect any abnormal behavior at all. Some attributes of field1 have more cyclical and predictable behavior than others, which makes them easier to detect abnormal behavior.
I'm finding that the more data the model takes in, the lower the severity the anomalies are scored with. I know this is just the nature of machine learning, but I need some way to continuing alerting on those events with the same level of severity. Is there anything I can do to hold this constant so they are not in the future disregarded as anomalies entirely?
Not to get too philosophical, but if you have a situation where the data's behavior is truly predictable like this, then Anomaly Detection isn't the right solution here because anomalies are, by definition, surprise events. So, maybe Just define a traditional alert?
Thank you for answering my question. I too was wondering if Anomaly Detection was the right solution for the case. Is there any other alerting system you might recommend for detecting an unexpected lack of data or a zero count then? I was finding that anomaly detection worked well for that but I'm not sure if it's a sustainable option over time.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
