Comparing the values of two indices(index1 vs forecast_index)

Elastic Stack Elasticsearch
1

Hello!

I have created a forecast for a specific product.
So, I want to compare the predicted values(forecast_index) with real values(index1) and then create an alert using watcher.
How can I return the data from indices in order to compare them in the watcher?

Thank you in advance!

/Angelos

2

Can you be more specific what you mean with comparing? Would two different queries (one for the first index, one for the second) be sufficient or not?

Please take some more time to explain your use-case properly, as it makes it much easier to help.

3

Hi Alexander,

For example the index with the real measurements is like this:

"hits": [
{
"_index": "Index1",
"_type": "test1",
"_id": "1",
"_score": 1,
"_source": {
"name": "john",
"product": 6
}
},
.
.
.
]

So I've made a forecast and the result is like this:

"hits": [
{
"_index": ".ml-anomalies-shared",
"_type": "doc",
"_id": "forecast_product_16546143111111",
"_score": 1,
"_source": {
"job_id": "...",
"forecast_id": "...",
"result_type": "model_forecast",
"bucket_span": ...,
"detector_index": 0,
"timestamp": 111111111111,
"partition_field_name": "...",
"partition_field_value": "...",
"model_feature": "'maximum value by person'",
"forecast_lower": 6,
"forecast_upper": 8,
"forecast_prediction": 7
}
},
]

Finally I want to compare the product with forecast_prediction and create an alert. If the forecast_prediction is higher than product send an email to me.
How can I compare these two measurements in a watcher? Or is there any other way I could do that?

Thank you very much in advance!

/Angelos

1 Like
4

this sounds so far as if you could use a chain input, that executes two queries and the have a script condition that checks if those values are different.

--Alex

5

Thank you Alexander!

I just did that but now I have problem on looping through 'hits'. My watcher is like this:

{
"trigger": {
"schedule": {
"interval": "30s"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
".ml-anomalies-"
],
"types": [],
"body": {
"size": 0,
"stored_fields": [
""
],
"script_fields": {},
"_source": [
"timestamp",
"partition_field_value",
"forecast_prediction"
],
"query": {
"bool": {
"must": [
{
"match": {
"result_type": "model_forecast"
}
},
{
"match": {
"job_id": "forecast_product"
}
},
{
"match": {
"partition_field_value": "Product1"
}
}
]
}
}
}
}
}
}
},
{
"second": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"products*"
],
"types": [],
"body": {
"size": 0,
"stored_fields": [
"*"
],
"script_fields": {},
"_source": [
"value-product"
],
"query": {
"bool": {
"must": [
{
"match": {
"product": "Product1"
}
}
]
}
}
}
}
}
}
}
]
}
},
"condition": {
"script": {
"source": "for (int i = 0; i < ctx.payload.second.hits.total; ++i) { if (ctx.payload.first.hits.hits[i]._source.forecast_prediction > ctx.payload.second.hits.hits[i]._source.value-product) { return true;}}",
"lang": "painless"
}
},
"actions": {
"my-logging-action": {
"logging": {
"level": "warn",
"text": "Logging..."
}
}
}
}

When I simulate I have:
"illegal_argument_exception","reason":"Variable [load] is not defined."}},"status":500}"}

So I assume is something wrong with the for loop.
Can you help me?

/Angelos

1 Like
6

Hello!

I think I found the problem. In my index the the product value is written like this "value-product". So in if statement the "dash" cannot be read.
Is there any way to solve this issue with the dash "-" in order to read the "ctx.payload.second.hits.hits[i]._source.value-product"?

/Angelos

1 Like
7

try ctx.payload.second.hits.hits[i]._source["value-product"]

You may want to rehink if it is feasible to compare each hit of one index with each hit of the other index. This will not work if one index has millions of items (which might not be a concern in your use-case, just something to keep in mind).

8

It works fine! I had to modified it a bit like this:
ctx.payload.second.hits.hits[i]._source['value-product']

Thank you very much Alex!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.