# Separate index for every aggs bucket

**URL:** <https://discuss.elastic.co/t/separate-index-for-every-aggs-bucket/247661>\
**Category:** Elasticsearch\
**Created:** [September 6, 2020, 8:54am UTC](https://discuss.elastic.co/t/separate-index-for-every-aggs-bucket/247661 "2020-09-06T08:54:31Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![bornatalebi](https://avatars.discourse-cdn.com/v4/letter/b/e68b1a/32.png) [@bornatalebi](https://discuss.elastic.co/u/bornatalebi)\
**Post date:** [September 6, 2020, 8:54am UTC](https://discuss.elastic.co/t/separate-index-for-every-aggs-bucket/247661/1 "2020-09-06T08:54:31Z")

</div>

Hi, I want to create a separate index for every aggs buckets. [here](https://gist.github.com/bornatalebi/afb0dc4f77020e0835ead87cf2f21eee) is my watcher.  
I know I have to use "foreach" for my problem but can't figure out what value should I give it.  
thanks.

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [September 6, 2020, 9:37am UTC](https://discuss.elastic.co/t/separate-index-for-every-aggs-bucket/247661/2 "2020-09-06T09:37:45Z")

</div>

Your tranform will produce this payload, so when you refer `ctx.payload.aggregations.groupbyhost.keys` it will produce following exception `specified foreach object was null: [ctx.payload.aggregations.device_name.keys]`  
there no aggregations.device\_name.keys under payload ....  
Please review your tranform script.

What is your target ? is to get for each device :

- Number of documents

- Array of users\_name

- Array of source\_ip  
Or something specific ?

---

<div class="post-metadata">

**Author:** ![bornatalebi](https://avatars.discourse-cdn.com/v4/letter/b/e68b1a/32.png) [@bornatalebi](https://discuss.elastic.co/u/bornatalebi)\
**Post date:** [September 6, 2020, 9:57am UTC](https://discuss.elastic.co/t/separate-index-for-every-aggs-bucket/247661/3 "2020-09-06T09:57:30Z")

</div>

Thanks for your reply,  
Yes I want to create an index for each device containing the Number of documents, Array of users\_name and Array of source\_ip.  
What should i replace `ctx.payload.aggregations.groupbyhost.buckets.keys` with? If i use something like `ctx.payload.host` it will only add host.name to index.

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [September 6, 2020, 10:20am UTC](https://discuss.elastic.co/t/separate-index-for-every-aggs-bucket/247661/4 "2020-09-06T10:20:37Z")

</div>

I would suggest you try [composite aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html) to get a simplified output index  
I'm not sure if that will help you, otherwise you need to review your tranform to produce the correct array you need

```
PUT _watcher/watch/ciscoioswatcher
{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "indices": [
          "filebeat-*"
        ],
        "body": {
          "size": 0,
          "query": {
            "bool": {
              "must": {
                "match": {
                  "event.code": {
                    "query": "LOGIN_FAILED"
                  }
                }
              },
              "filter": {
                "range": {
                  "@timestamp": {
                    "gte": "now-5m"
                  }
                }
              }
            }
          },
          "aggs": {
            "composite_buckets": {
              "composite": {
                "sources": [
                  {
                    "device_name": {
                      "terms": {
                        "field": "DevName"
                      }
                    }
                  },
                  {
                    "source_ip": {
                      "terms": {
                        "field": "source.ip"
                      }
                    }
                  },
                  {
                    "user_name": {
                      "terms": {
                        "field": "user.name"
                      }
                    }
                  }
                ]
              }
            }
          }
        }
      }
    }
  },
  "actions": {
    "index_payload": {
      "foreach": "ctx.payload.aggregations.composite_buckets.buckets",
      "max_iterations": 100,
      "index": {
        "index": "outputindex"
      }
    }
  }
}
```

---

<div class="post-metadata">

**Author:** ![bornatalebi](https://avatars.discourse-cdn.com/v4/letter/b/e68b1a/32.png) [@bornatalebi](https://discuss.elastic.co/u/bornatalebi)\
**Post date:** [September 6, 2020, 10:45am UTC](https://discuss.elastic.co/t/separate-index-for-every-aggs-bucket/247661/5 "2020-09-06T10:45:47Z")

</div>

> I would suggest you try [composite aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-composite-aggregation.html) to get a simplified output index  
> I'm not sure if that will help you, otherwise you need to review your tranform to produce the correct array you need

Thanks. I think I have to edit my transform script.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 4, 2020, 10:45am UTC](https://discuss.elastic.co/t/separate-index-for-every-aggs-bucket/247661/6 "2020-10-04T10:45:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.