Visualizing scripted metric aggregations in Kibana

[sean1010]

Hello,

I have some scripted metric aggregations in Elasticsearch that return some custom metrics from me. I'd like to add the ability to add this as a standard metric aggregation in my Kibana, but I don't see an easy way to do that. I've looked a little bit into scripted fields, but I don't think I can achieve the same results as in a scripted metric agg. (i.e. I need access to params, _source, etc.). Can anyone suggest a way to achieve this, or is there an alternate approach I'm missing?

Thanks in advance.

[flash1293 (Joe Reuter)]

Hi @sean1010,

unfortunately that's currently not possible at the moment. Scripted fields can only access one document at a time, so you can't create new aggregations spanning multiple documents with them. There is an open issue in the Github repository tracking the feature request: https://github.com/elastic/kibana/issues/2646

You can use them when creating vega visualizations because you can freely specify the Elasticsearch query there, but it comes with the cost of also having to define the chart itself in vega which is a whole separate language to define visualizations.

[Hendrik_Muhs]

Another alternative: you can create a transform and visualize the ouput of it in kibana.

Transform supports scripted metric aggregations, see examples.

Transforms comes for the price of extra storage but reduces load at query time as you do not have to re-run aggregations which might be expensive.

[sean1010]

Hi Hendrik, thanks for the reply. Are transforms similar to a pre-aggregation? Every time I visualize my transformation, I want to ensure the results are essentially computed on-the-fly so that I have access to the most recent metrics. Are transformations completed every time I view a Kibana visualization that uses that index? Is there a way I can achieve this?

I'm also wondering if we can only use transformations by creating a new index. Can I add to an existing index instead?

Thanks in advance.

[Hendrik_Muhs]

Yes, transforms are a form of pre-aggregation. In a nutshell a transform is a background task that runs search queries and writes the results into an index.

There are 2 types of transforms: "batch", which is a 1 time run, and "continuous" which continuously updates the data. The backgrond task periodically checks for updates and updates the required documents in the transformed index. The continuous mode requires a time-based index, so a field in the source data that contains a (real-time) timestamp. The update frequency can be adjusted using the frequency setting. Visualizations can be created on the transformed index while the query should run faster as the data is pre-computed. Have you checked our examples?

Its possible to create the dest index yourself, e.g. to tweak the number of shards, adjust the mappings, etc. Its also possible to write into an existing index. However this option has to be taken with care. You should not use the same index for the source and for the destination and mappings must be correctly configured.