# How can I reindex the results of an aggregation into another index? **URL:** \ **Category:** Elasticsearch\ **Tags:** vega\ **Created:** [July 16, 2021, 12:22pm UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896 "2021-07-16T12:22:04Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Vaishnavi\_Nandakumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaishnavi_nandakumar/32/89303_2.png) [@Vaishnavi\_Nandakumar](https://discuss.elastic.co/u/Vaishnavi_Nandakumar)\ **Post date:** [July 16, 2021, 12:22pm UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/1 "2021-07-16T12:22:04Z")
Hello, I'm using the 7.7.0 version of Elasticsearch. I have an aggregation query that works on the ingested data to produce the output but I am not able to re-index it. **Data Context** The given dataset has around 500,000 records consisting of Incoming File ID, Outgoing File ID, combination of both ID’s and the time elapsed during the transaction of both ID’s over a year. There can be multiple instances where both the File ID’s combinations (IN\_OUT\_ID) are repeated on different days. The aggregation query I have groups these records based on IN\_OUT\_ID and takes the document with the smallest Outgoing ID. I have used top hits aggregations to get the document source but when I tried reindexing with the query given below, all of the documents are being ingested into the destination index instead of just the filtered ones. ```auto POST /_reindex { "source": { "index": "src-new-data", "aggs": { "groupbyID": { "terms": { "field": "IN_OUT_ID", "order": { "lowest_score": "asc" } }, "aggs": { "lowest_score": { "min": { "field": "OUT_FILE_ID" } }, "lowest_score_top_hits": { "top_hits": { "size": 1, "sort": [ { "OUT_FILE_ID": { "order": "asc" } } ] } } } } } }, "dest": { "index": "dest-filter-data" } } ``` I want to visualize a histogram of the elapsed time from the filtered documents. If no other approach can be used for reindexing, is there any other way I can create the visualization based on the data context given? I have tried using Vega but I am not able to access the elapsed time value from the top hits bucket aggregation. Thank You ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [July 16, 2021, 2:12pm UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/2 "2021-07-16T14:12:13Z")
Have a look at [transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html). However top hits is not supported in transform, but you can use a `scripted_metric` aggregation, similar to this [example](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-painless-examples.html#painless-top-hits). ---
**Author:** ![Vaishnavi\_Nandakumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaishnavi_nandakumar/32/89303_2.png) [@Vaishnavi\_Nandakumar](https://discuss.elastic.co/u/Vaishnavi_Nandakumar)\ **Post date:** [July 17, 2021, 7:00pm UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/3 "2021-07-17T19:00:09Z")
Hi Hendrik, I checked out the link but I am not able to use \_transform as I'm using an earlier version of Elasticsearch. Is there any other approach I can use to either : a. Reindex from the top hits aggregations b. Use Vega to visualize top hits aggregations. ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [July 19, 2021, 6:47am UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/4 "2021-07-19T06:47:09Z")
> [@Vaishnavi\_Nandakumar](#): > > I checked out the link but I am not able to use \_transform as I'm using an earlier version of Elasticsearch. Transform has been made GA(General Availability) in elasticsearch as of version 7.5, it's predecessor (beta program) has been available since 7.2. Are you using the OSS distribution of elasticsearch? If not, in order to use transform you need to [activate the basic license](https://www.elastic.co/guide/en/elasticsearch/reference/current/start-basic.html). This license comes _free of charge_. Transform has been significantly improved after 7.7, however what I proposed should work in 7.7, too. > [@Vaishnavi\_Nandakumar](#): > > a. Reindex from the top hits aggregations Re-index can not write aggregation results into the destination index. > [@Vaishnavi\_Nandakumar](#): > > Use Vega to visualize top hits aggregations. I am not a Vega expert, but this seems possible to me. How many results are you expecting, how many do you want to show? If you are ok with the "top" top\_hits, this might work. ---
**Author:** ![Vaishnavi\_Nandakumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaishnavi_nandakumar/32/89303_2.png) [@Vaishnavi\_Nandakumar](https://discuss.elastic.co/u/Vaishnavi_Nandakumar)\ **Post date:** [July 19, 2021, 10:38am UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/5 "2021-07-19T10:38:58Z")
Hello, I'm using Open Distro for Elasticsearch. I tried to activate the basic license but I'm getting an error. ```auto { "error" : { "root_cause" : [ { "type" : "parse_exception", "reason" : "request body is required" } ], "type" : "parse_exception", "reason" : "request body is required" }, "status" : 400 } ``` > [@Hendrik\_Muhs](#): > > How many results are you expecting, how many do you want to show? If you are ok with the "top" top\_hits, this might work. The top hits aggregation is just producing 1 document per bucket. I want to access the value of a elapsed time field, within the source of that document. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 19, 2021, 10:57am UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/6 "2021-07-19T10:57:01Z")
You need to download the official version of elasticsearch if you want to use those features are they are not available within any other distribution. ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [July 19, 2021, 11:05am UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/7 "2021-07-19T11:05:30Z")
> [@Vaishnavi\_Nandakumar](#): > > I'm using Open Distro for Elasticsearch. I tried to activate the basic license but I'm getting an error. ok, in this case there is unfortunately no transform and no other basic licensed features. > [@Vaishnavi\_Nandakumar](#): > > The top hits aggregation is just producing 1 document per bucket. I want to access the value of a elapsed time field, within the source of that document. I actually meant how many buckets are you expecting. Do you want to show all buckets in 1 page? The reason for computing aggregations offline with e.g. transform is usually: - the number of buckets is large, like 10k or more (the limit can be lifted, but performance is another reason, offline pre-aggregation avoids running expensive queries) - you want to further analyze on top of the results, e.g. to answer questions like "average duration over all transactions" - you want to feed the results into another process like building a machine learning model ---
**Author:** ![Vaishnavi\_Nandakumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaishnavi_nandakumar/32/89303_2.png) [@Vaishnavi\_Nandakumar](https://discuss.elastic.co/u/Vaishnavi_Nandakumar)\ **Post date:** [July 19, 2021, 11:16am UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/8 "2021-07-19T11:16:28Z")
> [@Hendrik\_Muhs](#): > > ok, in this case there is unfortunately no transform and no other basic licensed features. Alright. I saw Ingest Pipelines can also be used for reindexing. Do you think a painless script can be used to filter out the documents? > [@](#): > > I actually meant how many buckets are you expecting. Do you want to show all buckets in 1 page? Ideally, yes. But I understand the limitations and I just want to see if the data could be visualized. ---
**Author:** ![Vaishnavi\_Nandakumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaishnavi_nandakumar/32/89303_2.png) [@Vaishnavi\_Nandakumar](https://discuss.elastic.co/u/Vaishnavi_Nandakumar)\ **Post date:** [July 19, 2021, 11:17am UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/9 "2021-07-19T11:17:21Z")
Oh I didn't know about this. Thank you. ---
**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\ **Post date:** [July 19, 2021, 12:13pm UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/10 "2021-07-19T12:13:34Z")
> [@Vaishnavi\_Nandakumar](#): > > Alright. I saw Ingest Pipelines can also be used for reindexing. Do you think a painless script can be used to filter out the documents? Ingest pipelines and the contained ingest processors can only take 1 document manipulate it and output 1 document, basically a "map" type of operation. They can not "reduce"[1], meaning you can not combine a set of documents. Aggregations are conceptually such a "reducer". I don't think re-index or ingest help you in this case. Vega will let you run and visualize/print aggregations, but might be limited to a small set. The only other option - if an upgrade of the cluster / switch to the official Elastic elasticsearch distribution[2] is not possible - is implementing your requirement in code, e.g. a python script that pulls the data and writes it back into another index. [1](Some rare special cases which are of type "reduce" are possible though with ingest pipelines, e.g. you can "enrich" one document with another) [2]Opendistro is a fork of elasticsearch not supported by Elastic ---
**Author:** ![Vaishnavi\_Nandakumar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaishnavi_nandakumar/32/89303_2.png) [@Vaishnavi\_Nandakumar](https://discuss.elastic.co/u/Vaishnavi_Nandakumar)\ **Post date:** [July 19, 2021, 12:58pm UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/11 "2021-07-19T12:58:58Z")
Got it. I'm currently limited to the platform I'm working with so I can't change that. But thank you for your response. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 16, 2021, 12:59pm UTC](https://discuss.elastic.co/t/how-can-i-reindex-the-results-of-an-aggregation-into-another-index/278896/12 "2021-08-16T12:59:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.