How to define tags and ignore_failure on Java API Client PutPipelineRequest

[afzm4 (Andrew Floyd)]

We are in the process of upgrading to the new Java API client and are running into some problems regarding using PutPipelineRequest. This is how we're defining a PutPipelineRequest:

PutPipelineRequest putPipelineRequest = new PutPipelineRequest.Builder()
               .id(pipelineName)
               .withJson(new StringReader(pipelineDefinition))
               .build();

With the an example pipelineDefinition of something like this:

{
  "description": "extracts the birthday month from the birth_date",
  "processors": [
   {
      "script": {
        "lang": "painless",
        "source": "example script here",
        "tag": "birth_month",
        "ignore_failure": true
      }
    }
  ]
}

We are finding that while this works when we exclude tag and ignore_failure from the request, it fails due to a deserialization error like the following when we include it in the script processor like above:

co.elastic.clients.json.JsonpMappingException: Error deserializing co.elastic.clients.elasticsearch._types.InlineScript: Unknown field 'tag' (JSON path: processors[0].script.tag) (line no=1, column no=681, offset=680)

Is there something we're missing here/doing incorrectly? Or is it possible that the new Java API Client doesn't support setting those fields? According to this documentation, its appears as though they should be able to be defined in the script processor definition. Any help would be appreciated. Thanks!

[dadoonet (David Pilato)]

Welcome!

I agree. The ScriptProcessor builder seems to be missing some options.

For example, ignore_failure can not be set:

client.ingest().putPipeline(pr -> pr
        .id("my-pipeline")
        .processors(p -> p
            .script(s -> s
                    .inline(is -> is
                            .source("ctx.foo = 'bar'")
                            .lang("painless"))
                            // This is not available
                            .ignoreFailure(true)
            )
        )
);

But it can with other processors like the set processor.

client.ingest().putPipeline(pr -> pr
        .id("my-pipeline")
        .processors(p -> p
            .set(s -> s
                    .field("foo")
                    .value(JsonData.of("bar"))
                    // This is available
                    .ignoreFailure(true)
            )
        )
);

I can see that there's an opened issue about this:

Script Processor's Builder class does not support most parameters.

[afzm4 (Andrew Floyd)]

Ah okay. Glad to know it wasn't something I was missing. I should be able to work around the missing functionality for now by updating those fields directly in the kibana stack management tool. But definitely something that would be useful to define within the inline script processor when building it in the Java API.

[ltrotta (Laura Trotta)]

Hey! Thank you for reporting this issue, we're currently working on it and it will be fixed soon