AnomalyDetection: Last bucket is not processed (bug?)

[eudaco (Eugene)]

Hello everybody. I have faced, as it seems to me, with strange behavior of anomaly detection jobs.
The last bucket is not processed when the model_plot_config is enabled.

For example, when I use a bucket size "1d", then I always get the analysis only the day before yesterday.

If I just disable model_plot_config, and the work starts working correctly.
In Job Management table, in both cases, I see the correct number of processed records, but this is not shown correctly in the chart (in case if model_plot is enabled).
Also, it doesn't matter what mode I run the datafeed (to end time or Real-time search).

I found this problem on the version 7.10.0. Also after upgrading to 7.12.0 nothing has changed.

I have already tried different configurations with frequency, bucket_span, query_delay... but the behavior is always the same.

Below is a very simplified example of a job and data to reproduce:

Job config:

    {
      "job_id": "events_job",
      "description": "Events Job",
      "analysis_config": {
        "bucket_span": "1d",
        "detectors": [
          {
            "function": "sum",
            "field_name": "Events",
            "detector_description": "Events Sum"
          }
        ],
        "influencers": []
      },
      "analysis_limits": {
        "model_memory_limit": "11MB"
      },
      "data_description": {
        "time_field": "Date",
        "time_format": "epoch_ms"
      },
      "model_plot_config": {
        "enabled": true,
        "annotations_enabled": true
      },
      "model_snapshot_retention_days": 10,
      "daily_model_snapshot_retention_after_days": 1,
      "results_index_name": "custom-events_job",
      "allow_lazy_open": true
    }

Datafeed config:

    {
      "query_delay": "1h",
      "query": {
        "match_all": {}
      },
      "frequency": "60m",
      "indices": [
        "events"
      ],
      "scroll_size": 1000,
      "delayed_data_check_config": {
        "enabled": true
      },
      "job_id": "events_job",
      "datafeed_id": "datafeed-events_job"
    }

Example source data for index:

    [
      {
        "Date": "2021-03-16",
        "Events": 5
      },
      {
        "Date": "2021-03-17",
        "Events": 10
      },
      {
        "Date": "2021-03-18",
        "Events": 5
      },
      {
        "Date": "2021-03-19",
        "Events": 10
      },
      {
        "Date": "2021-03-20",
        "Events": 5
      }
    ]

Result with enabled model_plot (with 4 processed buckets):

Result with disabled model_plot (with 5 processed buckets)::

I have been looking for similar cases in the community for a long time, but nobody seems to have come across this.
I would be glad for any help or idea. Have a nice day!

[dmitri (Dimitris Athanasiou)]

Hi Eugene,

Welcome to the elastic community! Thank you for posting this question about our anomaly detection product.

I believe what you describe is expected behaviour. However, I understand it seems odd. So let me try to explain as well as suggest a way to verify this is indeed what's going on.

Anomaly detection has a concept of interim results. In few words, results can be produced for a bucket which has not completed yet. For example, if you had a count detector and a bucket span of 1 hour, you could produce interim results at the half-hour mark into the latest bucket and see that some things already seem anomalous (e.g. the count has already grown much greater to the typical value, etc.).

When a job is run with a datafeed, interim results are produced automatically. Interim results enable early alerting as you don't have to wait for the end of the bucket to get notified for an anomaly happening.

Now let's discuss how the UI draws the data in the single metric viewer. When model plots are enabled, it draws the time series using the model plot data. This is to ensure consistency. When model plots are disabled, it draws looking at the source index.

So why does it look different? Because model plot data is only produced after we've seen the whole bucket, i.e. no model plots are produced for interim results. So, when you disable model plot data you can see the last interim bucket whereas when model plot is enabled you can't.

A way to verify there is no other difference would be to look at the results using the APIs directly.
You can call the Get Buckets API on the job with model plots enabled and the job with model plots disabled. Make sure to add the include_interim=true flag. I would expect the results are same.

I hope this has explained why you see the discrepancy.

Happy anomaly hunting!

Kind regards,
Dimitris

[eudaco (Eugene)]

Thanks for your reply @dmitri

Now I understand what is the point.
It would be great If it would be possible to set include_interim the kibana interface.

[dmitri (Dimitris Athanasiou)]

You're welcome @eudaco. l'll transfer your suggestion and the general issue to the team. Have a great April!

[Peter_Harverson (Pete Harverson)]

@eudaco I work on the Kibana ML UI, and I just wanted to add to the explanation that @dmitri has provided above, to say that when an interim anomaly record has been found, the chart in the Single Metric Viewer will plot the anomalous point in the chart using the actual value of that interim result even before the model plot data for that new point has been written. This ensures the interim result is shown in the chart, as well as being displayed in the anomalies table below, such as in the example below:

If the last bucket is not anomalous, then as @dmitri described above, the point will only be plotted in the chart when we've seen data for the complete bucket.

Hope this example helps.
Pete