# Dynamic scripting with painless, on\_failure, ingest pipelines **URL:** \ **Category:** Elasticsearch\ **Created:** [February 24, 2017, 3:28am UTC](https://discuss.elastic.co/t/dynamic-scripting-with-painless-on-failure-ingest-pipelines/76341 "2017-02-24T03:28:29Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![mattatriactenv](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mattatriactenv](https://discuss.elastic.co/u/mattatriactenv)\ **Post date:** [February 24, 2017, 3:28am UTC](https://discuss.elastic.co/t/dynamic-scripting-with-painless-on-failure-ingest-pipelines/76341/1 "2017-02-24T03:28:29Z")
We are on the AWS implementation on ES5. This has limitations which I am now trying to work around. The limitation I'd like help on relates to dynamic scripting - AWS appears to be only allowing Painless. This means that mustache is not available. (see announcement: [https://aws.amazon.com/about-aws/whats-new/2017/01/elasticsearch-5-now-available-on-amazon-elasticsearch-service/](https://aws.amazon.com/about-aws/whats-new/2017/01/elasticsearch-5-now-available-on-amazon-elasticsearch-service/)) I need help because we are using an "on\_failure" in our ingest pipelines. Essentially we are using a) filebeat to send logs to b) an ingest pipeline that c) uses GROK to parse it. We assume that our GROK syntax will not always be perfect and so, when a mistake happens, we want "on\_failure" to send the errant message to a new Index called Failed-IndexName. Elasic's excellent documentation shows how to do this: Source: [https://www.elastic.co/guide/en/elasticsearch/reference/current/handling-failure-in-pipelines.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/handling-failure-in-pipelines.html) ``` "on_failure" : [ { "set" : { "field" : "_index", "value" : "failed-{{ _index }}" } } ] ``` This is fantasic. Unfortunately the {{ }} in "failed-{{ \_index }}" is mustache, at least according to the error when I try to upload my pipeline. "script\_lang not supported [mustache]"},"header":{"processor\_type":"set"}} So I am trying Painless, which AWS says they do support for Dynamic Scripting. From my ingest pipeline: ``` "on_failure":[ { "set":{ "field":"_index", "value":{ "script":{ "type":"string", "script":{ "lang":"painless", "inline":"failed-doc['_index'].value" } } } } } ] ``` The error for on\_failure events is: Can not index event (status=400): {"type":"illegal\_argument\_exception","reason":"field [\_index] of type [java.util.HashMap] cannot be cast to [java.lang.String]"} I don't understand the error above any guidance would be appreciated. Thanks! ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [February 24, 2017, 9:27am UTC](https://discuss.elastic.co/t/dynamic-scripting-with-painless-on-failure-ingest-pipelines/76341/2 "2017-02-24T09:27:44Z")
The [`set` processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/set-processor.html) does not support custom scripts. For that you have to use the [`script` processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/script-processor.html). ---
**Author:** ![mattatriactenv](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mattatriactenv](https://discuss.elastic.co/u/mattatriactenv)\ **Post date:** [February 25, 2017, 5:50pm UTC](https://discuss.elastic.co/t/dynamic-scripting-with-painless-on-failure-ingest-pipelines/76341/3 "2017-02-25T17:50:45Z")
Thanks Yannick, It looks like the "on\_failure" block (even when specifying "Set") accepts 'mustache' as a way to reference variables - and I guess my question if I can use 'Painless' to do the same thing? The [documentation](https://www.elastic.co/guide/en/elasticsearch/reference/master/handling-failure-in-pipelines.html) says I can define "On\_Failure" as a "block" in my pipeline. The example uses 'mustache' to reference the value of the \_index name ( i.e., failed-{{ \_index }} ). ``` The following example defines an on_failure block on a whole pipeline to change the index to which failed documents get sent. { "description" : "my first pipeline with handled exceptions", "processors" : [...], "on_failure" : [ { "set" : { "field" : "_index", "value" : "failed-{{ _index }}" } } ] } ``` Since I get an error with the above documentation example due to 'mustache' not being allowed on the ES5 instance, I tried 'Painless'. I believe the syntax I want is something like, ``` "on_failure" : [ { "set" : { "field" : "_index", "value" : "failed-doc['_index'].value" } } ] ``` Unfortunately, ElasticSearch sees my 'Painless' syntax as a STRING. The end result is that my index is called "failed-doc['\_index'].value" instead of "failed-filebeat-2017-02-25". So I guess my question is can I use "Painless' to do the same thing that 'mustache' is doing in the above example and, if so, how. ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [February 26, 2017, 5:09pm UTC](https://discuss.elastic.co/t/dynamic-scripting-with-painless-on-failure-ingest-pipelines/76341/4 "2017-02-26T17:09:34Z")
The "set" processor just supports mustache templates. But you can use the "script" processor instead. `on_failure` is just a list of processors. You want to do something like the following: ```auto "on_failure" : [ { "script": { "lang": "painless", "inline": "ctx._index = 'failed-' + ctx._index" } } ] ``` ---
**Author:** ![mattatriactenv](https://avatars.discourse-cdn.com/v4/letter/m/7cd45c/32.png) [@mattatriactenv](https://discuss.elastic.co/u/mattatriactenv)\ **Post date:** [February 26, 2017, 8:55pm UTC](https://discuss.elastic.co/t/dynamic-scripting-with-painless-on-failure-ingest-pipelines/76341/5 "2017-02-26T20:55:45Z")
Thank you for the assist. Your insights into "on\_failure" and the "set" processor really helped me understand what is going on here, and your "script" processor modification to the "on\_failure" block works great. Using ingest node pipelines is new to me and this has been really helpful. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 26, 2017, 8:55pm UTC](https://discuss.elastic.co/t/dynamic-scripting-with-painless-on-failure-ingest-pipelines/76341/6 "2017-03-26T20:55:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.