# Foreach index action error

**URL:** <https://discuss.elastic.co/t/foreach-index-action-error/215107>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-alerting\
**Created:** [January 15, 2020, 9:49am UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107 "2020-01-15T09:49:33Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [January 15, 2020, 9:49am UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/1 "2020-01-15T09:49:33Z")

</div>

Hi,

As i was trying to index multiple hits using the foreach action and index action, i'm getting an error similar to [this post](https://discuss.elastic.co/t/transform-single-hit-into-multiple-documents-with-index-action/145010)  
When reading the [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-index.html) it says it support multiple documents and since \_doc field exists it will be indexed as a single document, I dont really understand what's causing the problem here since we hare using only few fields of the returned hits and not the whole document ( if this work as i think )

Here is the watch :

```auto
        POST _watcher/watch/_execute
    {
    "watch":{
      "trigger": {
        "schedule": {
          "cron": "* * * * * ?"
        }
      },
      "input": {
        "search": {
          "request": {
            "search_type": "query_then_fetch",
            "indices": [
              "filebeat-windows*"
            ],
            "rest_total_hits_as_int": true,
            "body": {
              "query": {
                  "bool": {
                  "filter": [
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-1m"
                        }
                      }
                    }
                  ],
                  "must_not":[{"term":{"windows.event.regleid.keyword":"-"}}]
                }
              }
            }
          }
        }
      },
      "actions": {
        "index_alert": {
          "transform": {
            "script": {
              "source": "return ['timestamp': ctx.payload.hits.hits.0._source['@timestamp'] ,'source.event.id':ctx.payload.hits.hits.0._id,'fields.clientid':ctx.payload.hits.hits.0._source.fields.clientid,'host.name':ctx.payload.hits.hits.0._source.host.name,'control': ctx.payload.hits.hits.0._source.windows.event.regleid ,'criticity': 'medium', 'operator':'-', 'perimeter': '-' , 'numticket':'-' , 'etat':'-' , 'rule.description': '-' ]",
              "lang": "painless"
            }
          },
          "foreach": "ctx.payload.hits.hits",
          "max_iterations": 99,
          "index": {
            "index": "alerts-windows"
          }
        }
      }
    }
    }

```

And the action error :

```auto
     "actions" : [
            {
              "id" : "index_alert",
              "type" : "index",
              "status" : "failure",
              "error" : {
                "root_cause" : [
                  {
                    "type" : "illegal_state_exception",
                    "reason" : "could not execute action [index_alert] of watch [_inlined_]. [ctx.payload._index] or [ctx.payload._doc._index] were set together with action [index] field. Only set one of them"
                  }
                ],
                "type" : "illegal_state_exception",
                "reason" : "could not execute action [index_alert] of watch [_inlined_]. [ctx.payload._index] or [ctx.payload._doc._index] were set together with action [index] field. Only set one of them"
              }
            }
          ]
        },
        "messages" : []
      }
    }

```

I have tried to implement Spinscale's script to delete \_index field but it failed since i have no idea of how this thing work 😅

Is there a way to do multiple docs indexing within a foreach condition ?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 15, 2020, 11:18am UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/2 "2020-01-15T11:18:18Z")

</div>

hey,

there is no need for `foreach`, as the `index` action itself can already index several documents, see [https://www.elastic.co/guide/en/elasticsearch/reference/7.5/actions-index.html#anatomy-actions-index-multi-doc-support](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/actions-index.html#anatomy-actions-index-multi-doc-support)

--Alex

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [January 15, 2020, 12:31pm UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/3 "2020-01-15T12:31:22Z")

</div>

Hi,

thanks for your answer, i see that the action index can indeed index several documents but this doesn't quite fit our use case since we use "ctx.payload.hits.hits.0" i struggle to find the right way to iterate over the hits and index them as we select and rename fields from the response.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 15, 2020, 4:27pm UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/4 "2020-01-15T16:27:56Z")

</div>

Sorry, I do not understand what you mean. Maybe you can explain this without mentioning any watcher functionality just to be sure everyone is on the same page.

--Alex

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [January 15, 2020, 4:31pm UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/5 "2020-01-15T16:31:52Z")

</div>

We would like to put the results of the query in another index,

actually the index action only take the first result and not all the hits...

that's why we were trying to use the foreach action.

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [January 15, 2020, 4:37pm UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/6 "2020-01-15T16:37:29Z")

</div>

To make it easier to understand :

- Query windows log index
- Create multiples docs from the query in another index

Thanks for your help, sorry if you have hard times understanding my english 😉

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [January 15, 2020, 4:50pm UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/7 "2020-01-15T16:50:50Z")

</div>

So this is possible, with the link written above in my previous post.

You can find an example of a script (the index\_transform.json one in the scripts directory) here: [https://github.com/elastic/examples/tree/master/Alerting/Sample%20Watches/port\_scan](https://github.com/elastic/examples/tree/master/Alerting/Sample%20Watches/port_scan)

One word of warning: This will not really work with a lot of results (like several 10k), because search after is not supported.

In order to continue I suggest you try to create a new watch, without a `foreach` part in the action and share the output of the execute watch action.

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [January 15, 2020, 5:12pm UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/8 "2020-01-15T17:12:18Z")

</div>

I'm sorry but i dont understand a lot of the index\_transform you sent to me

Dont worry about the amount of data i already saw your warning on previous posts and our use case will not be more than 100 results an hour.

In order to show you an example i edited two documents with the field **regleid** not equals to "-" so now the watch can match ...

My query has a total of **2** results which i would like to put in another index

```auto
  "actions": {
    "index_alert": {
      "transform": {
        "script": {
          "source": "return ['timestamp': ctx.payload.hits.hits.0._source['@timestamp'],'source.event.id':ctx.payload.hits.hits.0._id,'fields.clientid':ctx.payload.hits.hits.0._source.fields.clientid,'host.name':ctx.payload.hits.hits.0._source.host.name,'control': 'XXXX','criticity': 'medium','perimeter': 'XXXX','rule.description': 'XXX','XXX':ctx.payload.hits.hits.0._source.windows.event.regleid ]",
          "lang": "painless"
        }
      },
      "index": {
        "index": "alerts-windows"
      }
    }
  }
}

```

The result action

```auto
"actions" : [
        {
          "id" : "index_alert",
          "type" : "index",
          "status" : "success",
          "transform" : {
            "type" : "script",
            "status" : "success",
            "payload" : {
              "source.event.id" : "Aasjqm8Bc8wkC_ANKwFM",
              "fields.clientid" : "soc-m0",
              "perimeter" : "XXX",
              "criticity" : "XXX",
              "control" : "XXXX",
              "rule.description" : "XTXTXTXT",
              "host.name" : "HOST-XXXX",
              "Regle" : "XXX",
              "timestamp" : "2020-01-15T16:57:08.057Z"
            }
          },
          "index" : {
            "response" : {
              "created" : true,
              "result" : "created",
              "id" : "fQUkqm8BW9raCUYq-ncR",
              "version" : 1,
              "type" : "_doc",
              "index" : "alerts-windows"
            }
          }
        }
      ]
    },
    "messages" : []
  }

```

On this example we can see that depsite have **2** docs returned in the query the actions index only takes 1.  
I'm really trying to figure out how your script can solve my problem but i'm kinda having a though time doing so 🤯

---

<div class="post-metadata">

**Author:** ![grumo35](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grumo35/32/59451_2.png) [@grumo35](https://discuss.elastic.co/u/grumo35)\
**Post date:** [January 16, 2020, 1:42pm UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/9 "2020-01-16T13:42:52Z")

</div>

Here is my watch to index and filter multiple docs in one execution.

```auto
PUT foo/_doc/1
{
  "name" : {
    "first" : "Kobe",
    "last" : "Bryant"
  },
  "bar":"TO REMOVE"
}
PUT foo/_doc/2
{
  "name" : {
    "first" : "Stephen",
    "last" : "Curry"
  },
  "bar":"TO REMOVE"
}

PUT foo/_doc/3
{
  "name" : {
    "first" : "Dirk",
    "last" : "Nowitzki"
  },
  "bar":"TO REMOVE"
}

  POST _watcher/watch/_execute
{
  "watch" : {
  "trigger": {
    "schedule": {
      "interval": "2s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "foo"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
            "match_all": {}
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
      "index_payload": {
        "transform" : {
          "script" : 
          """
          def docs = [];
          for(item in ctx.payload.hits.hits) { 
                def d = item._source.clone(); 
 		d['end_timestamp'] = d['@timestamp']; 
 		d['@timestamp'] = ctx['trigger']['triggered_time'];
 		d.remove('bar');
 		docs.add(d);
          } 
                return ['_doc' : docs];
          """
        },
        "index": {
          "index": "reindex-test",
          "doc_type": "_doc"
        }
      }
    }
  }
}

```

To remove use d.remove and the name field.

Everything is in the watch here provided with examples. Thanks @spinscale & @hardbap

You may want to add this as an example since it's really hard to understand ( for me at-least ) and this is a very useful.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 13, 2020, 1:43pm UTC](https://discuss.elastic.co/t/foreach-index-action-error/215107/10 "2020-02-13T13:43:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.