# Re-indexing an aggregation for later use

**URL:** <https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [March 21, 2016, 3:45pm UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012 "2016-03-21T15:45:36Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![beckerdo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beckerdo/32/8355_2.png) [@beckerdo](https://discuss.elastic.co/u/beckerdo)\
**Post date:** [March 21, 2016, 3:45pm UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/1 "2016-03-21T15:45:36Z")

</div>

I am performing an aggregation on our company daily data stream in Elastic. I am bucketing the data by an "mid: field, and summing the "amount" field in the payload. It looks like this:

> ```
> {
> "aggs": {
> "tpv": {
> "terms": {
> "field": "payload.mid",
> "order" : { "total_volume" : "desc" }
> },
> "aggs": { 
> "total_volume": {
> "sum": {
> "field": "payload.usd_amt"
> }
> }
> }
> }
> }
> } 
> 
> ```

When I run the above aggregation, I see response data like this:

> {  
> "\_id": "cal\_tpv\_agg\_watch\_0-2016-03-21T13:20:56.372Z",  
> "result": {  
> "execution\_time": "2016-03-21T13:20:56.372Z",  
> "execution\_duration": 20,  
> "input": {  
> "aggregations": {  
> "tpv": {  
> "buckets": [  
> {  
> "doc\_count": 146,  
> "total\_volume": {  
> "value": 1559432  
> },  
> "key": "12347"  
> },  
> {  
> "doc\_count": 120,  
> "total\_volume": {  
> "value": 1239380  
> },  
> "key": "12352"  
> }  
> ]  
> }

I use an extract statement to put "aggregations.tpv.buckets" into ctx.payload.

I'd like to save the daily aggregation buckets data to another index (so that later we can roll up the data into weekly, monthly, or yearly amounts). I am doing this by using an index action to put the aggregation into another index.

Does anyone have an example of transforming the "ctx.payload.tpv.buckets" data to the payload "\_doc" field to take advantage of multi-document indexing specified in [Actions - Multi-doc support](https://www.elastic.co/guide/en/watcher/current/actions.html#anatomy-actions-index-multi-doc-support)

What's the best way of re-indexing aggregated data for later use?  
Thanks, beckerdo

---

<div class="post-metadata">

**Author:** ![beckerdo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beckerdo/32/8355_2.png) [@beckerdo](https://discuss.elastic.co/u/beckerdo)\
**Post date:** [March 21, 2016, 8:44pm UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/2 "2016-03-21T20:44:06Z")

</div>

Has anyone transformed an aggregation to a multi-document suitable input for an index action?

---

<div class="post-metadata">

**Author:** ![Chad\_Oliver](https://avatars.discourse-cdn.com/v4/letter/c/e47774/32.png) [@Chad\_Oliver](https://discuss.elastic.co/u/Chad_Oliver)\
**Post date:** [March 22, 2016, 4:45pm UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/3 "2016-03-22T16:45:52Z")

</div>

I have the same issue. I'm trying to use a Watcher that uses aggregation, but I want the results to be stored in a new Index as a multi-document. Are there examples how to do this?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 23, 2016, 8:39am UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/4 "2016-03-23T08:39:13Z")

</div>

Hey folks,

take this example for further testing

First, let's bulk index some docs

```auto
PUT /foo/bar/_bulk
{ "index" : { "_id" : "1" } }
{ "foo" : "bar" }
{ "index" : { "_id" : "2" } }
{ "foo" : "bar" }
{ "index" : { "_id" : "3" } }
{ "foo" : "baz" }
{ "index" : { "_id" : "4" } }
{ "foo" : "spam" }
{ "index" : { "_id" : "5" } }
{ "foo" : "spam" }
{ "index" : { "_id" : "6" } }
{ "foo" : "spam" }

```

After refresh, we should be able to search those and aggregate on them

```auto
GET /foo/bar/_search
{
  "size": 0,
  "aggs": {
    "the_foos": {
      "terms": {
        "field": "foo",
        "size": 10
      }
    }
  }
}

```

Let's get a watch up and running

```auto
PUT _watcher/watch/transform
{
  "input": {
    "search": {
      "request": {
        "indices": [
          "foo"
        ],
        "types": [
          "bar"
        ],
        "body": {
          "size": 0,
          "aggs": {
            "the_foos": {
              "terms": {
                "field": "foo",
                "size": 10
              }
            }
          }
        }
      }
    }
  },
  "trigger": {
    "schedule": {
      "interval": "1h"
    }
  },
  "actions": {
    "index_payload": {
      "transform": {
        "script": "return [_doc : ctx.payload.aggregations.the_foos.buckets]"
      },
      "index": {
        "index": "my-index",
        "doc_type": "my-type"
      }
    }
  }
}

```

No need to wait, execute!

```auto
POST _watcher/watch/transform/_execute

```

Knowing we ran the watch, let's check the index for new documents!

```auto
GET my-index/my-type/_search

```

On my 2.2.1 test installation this showed three documents... of course you can change the documents in your script transform to whatever you want, but this should be a start.

ID's are generated automatically here.

Hope this helps!

--Alex

---

<div class="post-metadata">

**Author:** ![beckerdo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/beckerdo/32/8355_2.png) [@beckerdo](https://discuss.elastic.co/u/beckerdo)\
**Post date:** [March 23, 2016, 2:43pm UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/5 "2016-03-23T14:43:07Z")

</div>

Brilliant. This worked perfectly for me. And I was able to write a second aggregation using data from the first aggregation.

Is there a way to do the transform (from buckets to \_doc) without the script? (For those servers that don't allow scripting.)

Thanks, Dan

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 23, 2016, 4:30pm UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/6 "2016-03-23T16:30:46Z")

</div>

Hey Dan,

not directly, but you can enable scripting specifically for watcher only by setting

```auto
script.engine.groovy.inline.elasticsearch-watcher_watch: on

```

Heads up: This setting is going to be renamed in 5.0.

--Alex

---

<div class="post-metadata">

**Author:** ![Chad\_Oliver](https://avatars.discourse-cdn.com/v4/letter/c/e47774/32.png) [@Chad\_Oliver](https://discuss.elastic.co/u/Chad_Oliver)\
**Post date:** [March 23, 2016, 5:44pm UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/7 "2016-03-23T17:44:36Z")

</div>

Alex...thank you. Your response on this thread was spot on and saved us. It would be helpful, if there was more documentation including examples online.

I have read through several Elasticserach ebooks, but I haven't come across any ebook that covers Watchers and Mult-docs.

-- Chad

---

<div class="post-metadata">

**Author:** ![ibelous](https://avatars.discourse-cdn.com/v4/letter/i/f14d63/32.png) [@ibelous](https://discuss.elastic.co/u/ibelous)\
**Post date:** [May 17, 2017, 12:09am UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/8 "2017-05-17T00:09:41Z")

</div>

not working for me with version 5.4

I copied and pasted provided example and got this error:

{  
"error": {  
"root\_cause": [  
{  
"type": "general\_script\_exception",  
"reason": "failed to compile script [ScriptException[compile error]; nested: IllegalArgumentException[Variable [\_doc] is not defined.];]"  
}  
],  
"type": "general\_script\_exception",  
"reason": "failed to compile script [ScriptException[compile error]; nested: IllegalArgumentException[Variable [\_doc] is not defined.];]"  
},  
"status": 500  
}

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [May 17, 2017, 6:52am UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/9 "2017-05-17T06:52:46Z")

</div>

Hey,

you can try putting `_doc` in ticks. Also, please open new threads for new issues.

Thanks a lot.

--Alex

---

<div class="post-metadata">

**Author:** ![ibelous](https://avatars.discourse-cdn.com/v4/letter/i/f14d63/32.png) [@ibelous](https://discuss.elastic.co/u/ibelous)\
**Post date:** [May 17, 2017, 8:33pm UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/10 "2017-05-17T20:33:14Z")

</div>

ticks around \_doc fixed it:

```
"transform": {
     "script": "return ['_doc' : ctx.payload.aggregations.the_foos.buckets]"
},

```

Thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:41pm UTC](https://discuss.elastic.co/t/re-indexing-an-aggregation-for-later-use/45012/11 "2017-07-06T13:41:58Z")

</div>