Dynamic scripting with painless, on_failure, ingest pipelines

[mattatriactenv (Matt)]

We are on the AWS implementation on ES5. This has limitations which I am now trying to work around.

The limitation I'd like help on relates to dynamic scripting - AWS appears to be only allowing Painless. This means that mustache is not available.

(see announcement: https://aws.amazon.com/about-aws/whats-new/2017/01/elasticsearch-5-now-available-on-amazon-elasticsearch-service/)

I need help because we are using an "on_failure" in our ingest pipelines. Essentially we are using a) filebeat to send logs to b) an ingest pipeline that c) uses GROK to parse it.

We assume that our GROK syntax will not always be perfect and so, when a mistake happens, we want "on_failure" to send the errant message to a new Index called Failed-IndexName.

Elasic's excellent documentation shows how to do this:
Source: https://www.elastic.co/guide/en/elasticsearch/reference/current/handling-failure-in-pipelines.html

  "on_failure" : [
    {
      "set" : {
        "field" : "_index",
        "value" : "failed-{{ _index }}"
      }
    }
  ]

This is fantasic. Unfortunately the {{ }} in "failed-{{ _index }}" is mustache, at least according to the error when I try to upload my pipeline.

"script_lang not supported [mustache]"},"header":{"processor_type":"set"}}

So I am trying Painless, which AWS says they do support for Dynamic Scripting. From my ingest pipeline:

  "on_failure":[  
      {  
         "set":{  
            "field":"_index",
            "value":{  
               "script":{  
                  "type":"string",
                  "script":{  
                     "lang":"painless",
                     "inline":"failed-doc['_index'].value"
                  }
               }
            }
         }
      }
   ]

The error for on_failure events is:
Can not index event (status=400): {"type":"illegal_argument_exception","reason":"field [_index] of type [java.util.HashMap] cannot be cast to [java.lang.String]"}

I don't understand the error above any guidance would be appreciated.

Thanks!

[ywelsch (Yannick Welsch)]

The set processor does not support custom scripts. For that you have to use the script processor.

[mattatriactenv (Matt)]

Thanks Yannick,

It looks like the "on_failure" block (even when specifying "Set") accepts 'mustache' as a way to reference variables - and I guess my question if I can use 'Painless' to do the same thing?

The documentation says I can define "On_Failure" as a "block" in my pipeline. The example uses 'mustache' to reference the value of the _index name ( i.e., failed-{{ _index }} ).

The following example defines an on_failure block on a whole pipeline to change the index to which failed documents get sent.

{
  "description" : "my first pipeline with handled exceptions",
  "processors" : [ ... ],
  "on_failure" : [
    {
      "set" : {
        "field" : "_index",
        "value" : "failed-{{ _index }}"
      }
    }
  ]
}

Since I get an error with the above documentation example due to 'mustache' not being allowed on the ES5 instance, I tried 'Painless'.

I believe the syntax I want is something like,

  "on_failure" : [
    {
      "set" : {
        "field" : "_index",
        "value" : "failed-doc['_index'].value"
      }
    }
  ]

Unfortunately, ElasticSearch sees my 'Painless' syntax as a STRING. The end result is that my index is called "failed-doc['_index'].value" instead of "failed-filebeat-2017-02-25".

So I guess my question is can I use "Painless' to do the same thing that 'mustache' is doing in the above example and, if so, how.

[ywelsch (Yannick Welsch)]

The "set" processor just supports mustache templates. But you can use the "script" processor instead. on_failure is just a list of processors. You want to do something like the following:

 "on_failure" : [
    {
      "script": {
        "lang": "painless",
        "inline": "ctx._index = 'failed-' + ctx._index"
      }
    }
  ]

[mattatriactenv (Matt)]

Thank you for the assist.

Your insights into "on_failure" and the "set" processor really helped me understand what is going on here, and your "script" processor modification to the "on_failure" block works great.

Using ingest node pipelines is new to me and this has been really helpful.