How to create an index_2 containing Index_1's aggregations results as fields?

Elastic Stack Kibana
1

How to create an index_2 containing Index_1's aggregations results as fields ?

Is it possible to use a command like this on kibana dev tools ?

PUT /index_2/doc_aggs_results_of_index1
{
field_aggs1: ...
field_aggs2: ...
{

For example:
Index_1 contains:
{
doc1:
name : Superman
city: New-York
date: 01-05-2015
}
{
doc2:
name : Batman
city: Los Angeles
date: 22-07-2018
}
{
doc3:
name : Superman
city: Sidney
date: 12-10-2017
}
{
doc4:
name : Batman
city: Paris
date: 17-02-2018
}

And I'd like to create index that aggregates or queries on index_1 to make index_2 :

Index_2 contains:
{
doc1:
name: Superman,
cities: [
{
city: New-York,
date: 01-05-2015
},
{
city: Sidney,
date: 12-10-2017
}]
},

{
doc2:
name: Batman,
cities: [
{
city: Los Angeles,
date: 22-07-2018
},
{
city: Paris,
date: 17-02-2018
}]
}

2

Hi, maybe dataframe transforms can help you with this: https://www.elastic.co/guide/en/elasticsearch/reference/7.6/ecommerce-transforms.html

3

It could have worked but unfortunatelly it seems that it only takes metric aggregations

4

Ah, I see. In this case you probably have to solve it outside of Elasticsearch- writing a script which executes the aggregation and feeds back the result into a separate index.

Maybe someone else can come up with a better solution. As this isn’t Kibana specific, it probably makes sense to repost this question in the Elasticsearch category.

5

This is not correct, transform supports other aggs as well, with scripted_metric - despite metric in the name - allows custom code which opens a lot of use cases.

We are expanding the list of supported aggs with every release. Which aggregations do you need?

6

I need terms aggregation but the dev tools manager answers to me:

"error": {
"root_cause": [
{
"type": "status_exception",
"reason": "Unsupported aggregation type [terms]"
}

7

thanks, we already have an enhancement request for terms.

In case its a small and known set of terms you can use a filter aggregation in the upcoming 7.7, see this example.

As already pointed out, with scripted_metric you could create a HashSet to map/reduce a terms frequency list.

I can not say when terms support in pivot will be available, but as this is a top ask: hopefully soon.

8

Filter doesn't adress my issue because it works by mentionning a specific field value, but I need to gather docs by field name.

9

For bucketing you use group_by, which supports grouping by term. For the collapsing part you can use a scripted_metric.

Here is an example configuration. The only thing missing according to your start post is dropping the field name.

POST _transform/_preview
{
  "source": {
    "index": "index-1"
  },
  "dest" : { 
    "index" : "index-2"
  },
  "pivot": {
    "group_by": {  
      "clientip": { "terms": { "field": "name" } }
      },
  "aggregations": {
    "all_docs": {
      "scripted_metric": {
        "init_script": "state.docs = []",
        "map_script": "state.docs.add(new HashMap(params['_source']))",
        "combine_script": "return state.docs",
        "reduce_script": "def docs = []; for (s in states) {for (d in s) { docs.add(d);}}return docs"
      }
    }
  }
}
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.