Is the document ID likely to change in Machine Learning result index

Elastic Stack Elasticsearch
1

Hi,

Is the document ID likely to change in Machine Learning result index ?

we have an anomaly alerting system, in the alerts I see document id that I can't find in the result index.

so I guess I'm right ?
if so then why is it changing ?

2

There are two reasons why ML results can be changed:

  1. Interim results being replaced by final results
  2. Renormalization

So it's true that you cannot rely on stable _ids in the ML results indices.

There's some information about renormalization here. And advice about ignoring interim results for alerting purposes here.

3

Thnaks.
I'm not sure what is the definition of interim results, I see:

If true , this is an interim result. In other words, the results are calculated based on partial input data.

what partial input data ? the historic data ? its already there, its part of the history and it won't be change, when and why the results calculated based on partial input data ?

4

There's a bit more about interim results in this blog:

As we receive interim data when using a datafeed, we also calculate interim results at the frequency interval. Interim results make adjustments for the fact that only partial data has been seen for the bucket. For functions such as min and max, interim results are very reliable. For functions such as sum, mean and count, then the reliability depends on the characteristics of the data. Adjustments for partial buckets with continuous data can be very accurate. However if your data time stamps are irregular and low volume, then it becomes difficult to assess how much of the bucket has been seen so confidence is less and interim anomalies may not be found (or in the case of low_count or low_sum functions may be inaccurate).

So basically you might get interim results if your datafeed frequency is smaller than you bucket span. Once all the data for a bucket has been seen those interim results are deleted and replaced with the final results for a bucket.

5

yes, this is the case.
thanks a lot for help :slight_smile:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.