Xpack anomaly detect theory

Elastic Stack Elasticsearch
1

Dear X-Pack developers,

I have already checked the open source code of Xpack for personal interest, I am not sure which kinds of algorithm is used for machine learning and I also found move average aggregation algorithm in core ES code but did not find any provoke from XPack.

Could you please kindly advice where I can find the algorithm in XPACK? Thank you very much.
I would be great if you can share some theory of Xpack anomaly detection.

2

Some videos that describe the math behind X-Pack Machine Learning:

2017 ElasticON:
https://www.elastic.co/elasticon/conf/2017/sf/machine-learning-and-statistical-methods-for-time-series-analysis

2018 ElasticON:
https://www.elastic.co/elasticon/conf/2018/sf/the-math-behind-elastic-machine-learning

3

thanks for your reply, I found the ml module is packed into C++ and watch the video, it looks like the anomaly score and forecast are used by different algorithm.

TScores::compute(jointProbabilityWeight, extremeProbabilityWeight,
minExtremeSamples, maxExtremeSamples, maximumAnomalousProbability,
p1, overallScore, overallProbability);

I am still not very clear what kind of algorithm is used to calculate the score and the probability. Is there any guide combined with the code analysis? or some docker image for that ML C++ source code environment which is used to debug more easily?

4

No, there is no code guide.

The probability calculation comes from measuring the observed sample against the probability model that's been constructed behind the scenes by X-Pack ML. The basis of the approach has been described in this academic paper: http://www.ijmlc.org/papers/398-LC018.pdf

The normalized anomaly scoring approach is described in a practical sense here: https://www.elastic.co/blog/machine-learning-anomaly-scoring-elasticsearch-how-it-works

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.