connect ElasticSearch script to Kibana dashboard

Elastic Stack Elasticsearch
1

I don't understand how to connect Elasticsearch Dev Tools scripts to Kibana dashboard.

Say that we have a nice script:

GET name_of_my_index/_search
{
  "size": 0,
  "aggs": {
    "tags_count": {
      "scripted_metric": {
        "init_script": "state.counts = [:]",
        "map_script": """
          some script here
          }
        """,
        "combine_script": "return state.counts",
        "reduce_script": """
          more cool scripting here
        """
      }
    }
  }
}

The script works for me in "Dev Tools", but I would also like to visualize the results of this script in Kibana (just a simple bar graph to show name:count pairs). How would I do that?

I was expecting that the search/script can be saved as some kind of field/index/query/script that I could then call in a Kibana visualization lens to display. But, it seems that things are not so simple and I'm a bit stuck here.

Any tip would be great. Thanks!

2

As a general rule you can't provide query dsl to kibana to use as the basis of a visualization.

Normally you'd use kibana lens to build a visualization and you'd configure the visualization to closely match what your is doing. You can even view the query dsl that Kibana is using and adjust the visualization until it does what your query dsl does.

That being said, scripted metrics have fallen out of favor in the Elasticsearch world and it is not possible to create a Kibana Lens visualization that leverages them. You could create a Vega visualization but that's got a very high barrier to entry.

So what options do you have?

  1. Attempt to get the same behavior using normal visualization primitives in Kibana Lens
  2. Use ES|QL as you can both query using ES|QL and you can use it as the basis for a Kibana Lens visualization Using ES|QL in Kibana | Elasticsearch Guide [8.18] | Elastic
  3. Use a Transform to perform the calculation and load the result into an index where it can be visualized after calculation Transforming data | Elasticsearch Guide [8.17] | Elastic
  4. Use a Vega visualization Custom visualizations with Vega | Kibana Guide [8.17] | Elastic

Id probably go the ES|QL route as if your use-case fits. If you can share the metric calculation you're doing (a long with a couple sample documents) I may be able to offer you an ES|QL query that does the same thing.

3

Hi, William.

Thanks for your reply.

Here are the calculations I'm doing:

GET my_index_0.0.22/_search
{
  "size": 0,
  "aggs": {
    "Total number of each component": {
      "scripted_metric": {
        "init_script": "state.counts = [:]",
        "map_script": """
          for (component in params._source.slowest_components_total) {
              state.counts[component] = state.counts.containsKey(component) ? state.counts[component] + 1 : 1;
          }
        """,
        "combine_script": "return state.counts",
        "reduce_script": """
          Map finalCounts = [:];
          for (state in states) {
            for (entry in state.entrySet()) {
              finalCounts[entry.getKey()] = finalCounts.containsKey(entry.getKey()) ?
                finalCounts[entry.getKey()] + entry.getValue() : entry.getValue();
            }
          }
          return finalCounts;
        """
      }
    }
  }
}

A sample document:

{
  "_index": "my_index_0.0.22",
  "_type": "_doc",
  "_id": "Mf-8W5UBwBI085POA_tP",
  "_version": 1,
  "_score": 1,
  "_source": {
    "dvideo_quality": "100",
    "feature": "ABC",
    "start_timestamp": "1665709037.75",
    "scenario_duration": 4.28,
    "query_start": "2022-10-14T02:57:17.750000Z",
    "query_stop": "2022-10-14T02:57:22.030000Z",
    "init_start": "2025-03-03T11:17:37.583979Z",
    "init_stop": "2025-03-03T11:17:58.118608Z",
    "buffer_start": "2025-03-03T11:18:11.110742Z",
    "buffer_stop": "2025-03-03T11:20:02.990229Z",
    "init_duration": 20.534629,
    "buffer_duration": 111.879487,
    "slowest_components_total": [
      "AAA",
      "AAA",
      "AAA",
      "AAA",
      "BBB",
      "BBB"
    ],
    "id": "e506d477-f138-4553-9b63-dc8a95d65b54",
    "timestamp": "2025-03-03T11:20:02.997252Z",
    "total_files_size": 113.30000000000001
  },
  "fields": {
    "start_timestamp": [
      "1665709037.75"
    ],
    "event_type": [
      "event"
    ],
    "dvideo_quality": [
      "100"
    ],
    "id": [
      "e506d477-f138-4553-9b63-dc8a95d65b54"
    ],
    "node": [
      "105"
    ],
    "cpu_usage": [
      0.028979167
    ],
    "timestamp": [
      "2025-03-03T11:20:02.997Z"
    ],
    "dvideo_quality.keyword": [
      "100"
    ]
  }
}

I simply want to count the number of occurrences of each component under slowest_componens_total, so I need an end count as "AAA":4, "BBB":2.

Any help would be great. Thanks a lot :slight_smile:

4

This ES|QL query should do this for you:

FROM test-index | 
KEEP slowest_components_total |
MV_EXPAND slowest_components_total | 
STATS count() BY slowest_components_total
5

That looks really simple. Thanks again... but I'm guessing that ES|QL is not available on Kibana 7.17, right? I can't seem to find it (only KQL and Lucene)

6

IIRC ES|QL was first introduced in 8.11, so no.

1 Like
7

If I find time today or tomorrow I'll try to do this in query dsl as well

8

Thank you :slightly_smiling_face: