Foreach index action error

[grumo35]

Hi,

As i was trying to index multiple hits using the foreach action and index action, i'm getting an error similar to this post
When reading the documentation it says it support multiple documents and since _doc field exists it will be indexed as a single document, I dont really understand what's causing the problem here since we hare using only few fields of the returned hits and not the whole document ( if this work as i think )

Here is the watch :

        POST _watcher/watch/_execute
    {
    "watch":{
      "trigger": {
        "schedule": {
          "cron": "* * * * * ?"
        }
      },
      "input": {
        "search": {
          "request": {
            "search_type": "query_then_fetch",
            "indices": [
              "filebeat-windows*"
            ],
            "rest_total_hits_as_int": true,
            "body": {
              "query": {
                  "bool": {
                  "filter": [
                    {
                      "range": {
                        "@timestamp": {
                          "gte": "now-1m"
                        }
                      }
                    }
                  ],
                  "must_not":[{"term":{"windows.event.regleid.keyword":"-"}}]
                }
              }
            }
          }
        }
      },
      "actions": {
        "index_alert": {
          "transform": {
            "script": {
              "source": "return [ 'timestamp': ctx.payload.hits.hits.0._source['@timestamp'] ,'source.event.id':ctx.payload.hits.hits.0._id,'fields.clientid':ctx.payload.hits.hits.0._source.fields.clientid,'host.name':ctx.payload.hits.hits.0._source.host.name,'control': ctx.payload.hits.hits.0._source.windows.event.regleid ,'criticity': 'medium', 'operator':'-', 'perimeter': '-' , 'numticket':'-' , 'etat':'-' , 'rule.description': '-'  ]",
              "lang": "painless"
            }
          },
          "foreach": "ctx.payload.hits.hits",
          "max_iterations": 99,
          "index": {
            "index": "alerts-windows"
          }
        }
      }
    }
    }

And the action error :

     "actions" : [
            {
              "id" : "index_alert",
              "type" : "index",
              "status" : "failure",
              "error" : {
                "root_cause" : [
                  {
                    "type" : "illegal_state_exception",
                    "reason" : "could not execute action [index_alert] of watch [_inlined_]. [ctx.payload._index] or [ctx.payload._doc._index] were set together with action [index] field. Only set one of them"
                  }
                ],
                "type" : "illegal_state_exception",
                "reason" : "could not execute action [index_alert] of watch [_inlined_]. [ctx.payload._index] or [ctx.payload._doc._index] were set together with action [index] field. Only set one of them"
              }
            }
          ]
        },
        "messages" : [ ]
      }
    }

I have tried to implement Spinscale's script to delete _index field but it failed since i have no idea of how this thing work

Is there a way to do multiple docs indexing within a foreach condition ?

[spinscale (Alexander Reelsen)]

hey,

there is no need for foreach, as the index action itself can already index several documents, see https://www.elastic.co/guide/en/elasticsearch/reference/7.5/actions-index.html#anatomy-actions-index-multi-doc-support

--Alex

[grumo35]

Hi,

thanks for your answer, i see that the action index can indeed index several documents but this doesn't quite fit our use case since we use "ctx.payload.hits.hits.0" i struggle to find the right way to iterate over the hits and index them as we select and rename fields from the response.

[spinscale (Alexander Reelsen)]

Sorry, I do not understand what you mean. Maybe you can explain this without mentioning any watcher functionality just to be sure everyone is on the same page.

--Alex

[grumo35]

We would like to put the results of the query in another index,

actually the index action only take the first result and not all the hits...

that's why we were trying to use the foreach action.

[grumo35]

To make it easier to understand :

• Query windows log index
• Create multiples docs from the query in another index

Thanks for your help, sorry if you have hard times understanding my english

[spinscale (Alexander Reelsen)]

So this is possible, with the link written above in my previous post.

You can find an example of a script (the index_transform.json one in the scripts directory) here: https://github.com/elastic/examples/tree/master/Alerting/Sample%20Watches/port_scan

One word of warning: This will not really work with a lot of results (like several 10k), because search after is not supported.

In order to continue I suggest you try to create a new watch, without a foreach part in the action and share the output of the execute watch action.

[grumo35]

I'm sorry but i dont understand a lot of the index_transform you sent to me

Dont worry about the amount of data i already saw your warning on previous posts and our use case will not be more than 100 results an hour.

In order to show you an example i edited two documents with the field regleid not equals to "-" so now the watch can match ...

My query has a total of 2 results which i would like to put in another index

  "actions": {
    "index_alert": {
      "transform": {
        "script": {
          "source": "return ['timestamp': ctx.payload.hits.hits.0._source['@timestamp'],'source.event.id':ctx.payload.hits.hits.0._id,'fields.clientid':ctx.payload.hits.hits.0._source.fields.clientid,'host.name':ctx.payload.hits.hits.0._source.host.name,'control': 'XXXX','criticity': 'medium','perimeter': 'XXXX','rule.description': 'XXX','XXX':ctx.payload.hits.hits.0._source.windows.event.regleid  ]",
          "lang": "painless"
        }
      },
      "index": {
        "index": "alerts-windows"
      }
    }
  }
}

The result action

"actions" : [
        {
          "id" : "index_alert",
          "type" : "index",
          "status" : "success",
          "transform" : {
            "type" : "script",
            "status" : "success",
            "payload" : {
              "source.event.id" : "Aasjqm8Bc8wkC_ANKwFM",
              "fields.clientid" : "soc-m0",
              "perimeter" : "XXX",
              "criticity" : "XXX",
              "control" : "XXXX",
              "rule.description" : "XTXTXTXT",
              "host.name" : "HOST-XXXX",
              "Regle" : "XXX",
              "timestamp" : "2020-01-15T16:57:08.057Z"
            }
          },
          "index" : {
            "response" : {
              "created" : true,
              "result" : "created",
              "id" : "fQUkqm8BW9raCUYq-ncR",
              "version" : 1,
              "type" : "_doc",
              "index" : "alerts-windows"
            }
          }
        }
      ]
    },
    "messages" : [ ]
  }

On this example we can see that depsite have 2 docs returned in the query the actions index only takes 1.
I'm really trying to figure out how your script can solve my problem but i'm kinda having a though time doing so

[grumo35]

Here is my watch to index and filter multiple docs in one execution.

PUT foo/_doc/1
{
  "name" : {
    "first" : "Kobe",
    "last" : "Bryant"
  },
  "bar":"TO REMOVE"
}
PUT foo/_doc/2
{
  "name" : {
    "first" : "Stephen",
    "last" : "Curry"
  },
  "bar":"TO REMOVE"
}

PUT foo/_doc/3
{
  "name" : {
    "first" : "Dirk",
    "last" : "Nowitzki"
  },
  "bar":"TO REMOVE"
}

  POST _watcher/watch/_execute
{
  "watch" : {
  "trigger": {
    "schedule": {
      "interval": "2s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "foo"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
            "match_all": {}
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
      "index_payload": {
        "transform" : {
          "script" : 
          """
          def docs = [];
          for(item in ctx.payload.hits.hits) { 
                def d = item._source.clone(); 
 		d['end_timestamp'] = d['@timestamp']; 
 		d['@timestamp'] = ctx['trigger']['triggered_time'];
 		d.remove('bar');
 		docs.add(d);
          } 
                return [ '_doc' : docs];
          """
        },
        "index": {
          "index": "reindex-test",
          "doc_type": "_doc"
        }
      }
    }
  }
}

To remove use d.remove and the name field.

Everything is in the watch here provided with examples. Thanks @spinscale & @hardbap

You may want to add this as an example since it's really hard to understand ( for me at-least ) and this is a very useful.