Index action - document construction

Elastic Stack Elasticsearch
1

I wanted to use index action.
The initial payload in the index action is quite a big one. I'd like to construct smaller doc to be inserted.

Actions like logging or email allow for using templates:

"logging" : {
        "text" : "executed at {{ctx.execution_time}}" 
}

It doesn't seem to be the case with index action. So I would probably need use transform ability it provides.

I see in the docs (https://www.elastic.co/guide/en/watcher/current/anatomy-transform.html) that there are two types (3 including 'chain') of transforms: script and search.

Wouldn't it be useful to be able to use templates to create my own doc?

For example by having dedicated field for all 'injected' attributes, like this:

"actions" : {
   "index_payload" : { 
     "transform: { 
     },
     "index" : {
       "index" : "my-index", 
       "doc_type" : "my-type" 
     },
     "doc": { 
        "executedAt" : "executed at {{ctx.execution_time}}",
        "field1": "some simple text with {{ctx.some_expresion}}"
    }  
  }
}

Come to think of it, a transformation of this kind (document builder, or something) would be even more useful.
At least I think so..... :smile:

What do you think?

Regards,
Karol

2

I like it! totally makes sense for the cases where you don't need control flow logic, in which a template is much more readable.

Thanks Karol for the suggestion!

3

On a similar note, If I do something like this to save a piece of data using the index action, it nests the results under "data": {"_value":

"actions": {
    "index_payload": {
        "transform": {
            "script" : "return ctx.payload.hits.hits*._source"
        },
        "index": {
            "index": "issues",
            "doc_type": "issue"
        }
    }
}

Produces:

 "_source":{
  "data" : {
    "_value" : [ {
      "@timestamp" : "2015-05-26T17:41:38.2002937Z",
      "_type" : "processor",
      "processor" : {
        "PercentProcessorTime" : 90,
      }
    } ]
  }

Is there a way to avoid this behavior? If I need to satisfy an existing model this is problematic.

4

If you return a map from the script you should be able to adhere to whatever model you need. See https://www.elastic.co/guide/en/watcher/current/anatomy-transform.html#anatomy-transform-script

5

I tried to use watcher in this way:

  1. initial payload is search request with some aggregations

  2. as a watcher action I wanted to index aggregation buckets in way "1
    bucket from search response results in 1 new indexed document"

but I got in trouble with mentioned "data": {"_value": structure... - which doesn't fit required mapping, I need unwrap this "data" field wrapper.
Is possible to create transform script which leads to required behavior - multiple new indexed documents?

6

Oh.. interesting. We currently don't support indexing multiple docs per a single action, but I love the idea. So for example, if the payload is an array, we can treat each element of the array as a document and bulk index it... nice one!

7

Thanks GaelTadh for your reply, that behavior would be great.

8

Patrik, please note that this functionality was added in beta 2:

https://www.elastic.co/guide/en/watcher/current/actions.html#anatomy-actions-index-multi-doc-support

It's a bit different than what is described above, but still supported - just put whatever array of docs under the _doc field in the payload