Check Forecast accuracy with scripted field

[CHP93 (Christoph)]

Hello community,

I have a quite challenging task and have not found a solution for my problem, yet.
I created a multi-metric anomalies detection machine learning job which is running a forecast as well. My aim is now to compare the forecast results with the actual results to get a sense of the accuracy and quality of the calculated forecast. In my knowledge the only possible way of solving this task would be to create a scripted field. I am not sure if it is already possible within Kibana to compare fields from different index pattern. I hope someone is able to help me with that challenge!

Thank you!

Christoph

[BenB196]

I think you might be able to achieve this, but it will be kind of weird.

You might be able to achieve this using transforms and the lookup runtime field.

Concept, use transforms with the source index containing the actual data. Add a lookup runtime field that searches against the forecast index to add the forecast data to the actual data document. Transform will then index this result into a new index/document which will contain all of the needed information where you can then perform the comparison.

Note: You need to do this round-about transform method because of the limitations with the lookup runtime field not being able to be aggregated against directly.

[CHP93 (Christoph)]

Hello Ben,

thank you for your answer! The past days I tried to implement your solution. I can transform the actual data over into a separate index. Unfortunately I am struggling with the second transform from the forecast index pulling the data over to the new index. Is that possible with a transform as well or do I need to implement a runtime field. How could such a runtime lookup look like? Maybe you can help me with that. Thank you!

Christoph

[BenB196]

Hmm, so I think you should only need one transform. In that transform you should use your "real" data index, then add a lookup runtime field that queries your "forecast" index.

Maybe something like;

PUT _transform/asdf
{
  "source": {
    "index": [
      "metrics-kubernetes.container-private.kubernetes.development"
    ],
    "runtime_mappings": {
      "ml_forecast": {
	  	"type": "lookup",
	  	"target_index": ".ml-anomalies-custom-kubernetes_container_resource_regressions_development",
	  	"input_field": "kubernetes.pod.name"
	  	"target_field": "kubernetes.pod.name",
	  	"fetch_fields": ["forecast_*"]
	  }
    }
  },
  "pivot": {
    "group_by": {
      "kubernetes.pod.name": {
        "terms": {
          "field": "kubernetes.pod.name"
        }
      },
      "kubernetes.container.name": {
        "terms": {
          "field": "kubernetes.container.name"
        }
      },
      "ml_forecast": {
        "terms": {
          "field": "ml_forecast"
        }
      }
    },
    "aggregations": {
      "kubernetes.container.cpu.usage.nanocores.avg": {
        "avg": {
          "field": "kubernetes.container.cpu.usage.nanocores"
        }
      }
    }
  },
  "dest": {
    "index": "asdf"
  },
  "sync": {
    "time": {
      "field": "@timestamp"
    }
  }
}

This is just a theoretical example, I'm not 100% sure it will work. Just more to demonstrate the idea.

Note, it looks like you can't add lookup runtimes via the Kibana UI, I opened this issue about it. So, you'll need to create the transform via the Elasticsearch API.