X-pack low document count

[palace]

Hello,

I have an ML model which looks at the low count of a particular job. The purpose of this ML job is to notify when a job has not not been initiated in a long time. For instance, if the count of jobs initiated was 0 in 10hrs, that would be an anomaly.

The job works fine, it is finding correct anomalies but is also skipping a good deal. Please refer to the image below.

I am wondering if there is a better way to set up this job or are these the best results?

I set my bucket span to 10hr, should I reduce it?

Thank you in advance.

[dkyle (David Kyle)]

HI,

From the image it looks like you have around 6 months of data and there was a huge jump in May which changed the model bound (the light blue plot). This has obviously had an affect on the model which may be why you don't see an anomaly on 18th June.

You are doing the right thing using the one sided low_count function but if your data is sparse or has missing records you may benefit from using the low_non_zero_count function which ignores gaps in the data.

Assuming this is a single metric job and it's quick to run I would create multiple jobs and experiment with different bucket spans. A 10 hr bucket span is longer than I would usually recommend try running the job with shorter bucket spans.

if the count of jobs initiated was 0 in 10hrs, that would be an anomaly.

If you specifically want to catch this case why not create a Watcher alert with a Date Histogram and alert when doc_count == 0.