Extract the results of a ML job

Elastic Stack Elasticsearch
1

Is there any way to extract the results of a ML job(i.e, the detected anomalies in input data) using languages like python with their Elasticsearch client?

2

All results from ML jobs are stored in the .ml-anomalies-* indices and therefore are accessible via query from any ES client

3

I used the following code to fetch the results for the anomaly jobs
doc = {
'size' : 10000,
'query': {
'match_all' : {}
}
}
res=es.search(index='.ml-anomalies-*', body=doc)

I got the following output:

{'_index': '.ml-anomalies-shared',
'_type': '_doc',
'_id': 'abcd_bucket_1560420000000_3600',
'_score': 1.0,
'_source': {'job_id': 'abcd',
'timestamp': 1560420000000,
'anomaly_score': 0.0,
'bucket_span': 3600,
'initial_anomaly_score': 0.0,
'event_count': 68,
'is_interim': True,
'bucket_influencers': ,
'processing_time_ms': 2,
'result_type': 'bucket'}

When I tried to convert the timestamp to date time format, I get the below error:
----> 1 dt=datetime.fromtimestamp(1560420000000)

ValueError: year 51417 is out of range

In what format does the .ml-anomalies-* index store the timestamp value? Is any formatting needed for the timestamp values returned to get the correct date and time?

4

Does this help?

Internally, dates are converted to UTC (if the time-zone is specified) and stored as a long number representing milliseconds-since-the-epoch.

You should be able to convert from that.

5

Thank you, I resolved the issue now. I just divided the timestamp by 1000 as it was in milliseconds

6

Notice that inside the result index, there are a variety of different documents, each with their own usefulness

result_type:bucket
result_type:record
result_type:influencer

More info here: https://www.elastic.co/blog/machine-learning-anomaly-scoring-elasticsearch-how-it-works

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.