Injest Pipeline for conditional statement used in Scripts

[Tomo_M (Tomohiro Mitani)]

@shivani_singh

if( ctx['responseBody'] != null && ctx.containsKey('results'))

The latter make no sense here if the former does not raise any error.

[shivani_singh]

@Tomo_M

e what you want to do, it is possible to create an array and add values in each iteration, then finally set the array to ctx['embedded_error_code'] .

As I said, responseBody contains different format of json data within so it might happen there is responseBody but not results.

[shivani_singh]

Now I'm renaming -

 {
      "rename": {
        "field": "responseBody.results",
        "target_field": "embedded_original_value_obj",
        "ignore_missing": true
      }
    },

So now I'm taking array results into my new field embedded_original_value_obj which has type: object in mapping.
It is working fine and I'm getting responseBody.results[0].statusCode value in embedded_original_value_obj.statusCode

Note : Both set and rename is working fine for me here, I'm using set in below example

[shivani_singh]

Hi @Tomo_M ,

I'm using set here.
1. set is working and I'm getting value in embedded_original_value_obj but 2. set isn't working. Any idea how can do this?

PUT _ingest/pipeline/apiproxy-embedded-copy
{
  "processors": [
    {
1.      "set": {
        "field": "embedded_original_value_obj",
        "copy_from": "responseBody.results",
        "ignore_empty_value": true
      }
    },
    {
2.      "set": {
        "field": "embedded_error_code",
        "copy_from": "embedded_original_value_obj.statusCode",
        "ignore_empty_value": true
      }
    }
]
}

[Tomo_M (Tomohiro Mitani)]

Rename looks good to achive your goal.

But as for set processors,

POST _ingest/pipeline/_simulate
{
  "docs": [
    {
      "_source":{
        "responseBody":{
          "results":{
            "statusCode": "200"
          }
        }
      }
    }
  ],
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "embedded_original_value_obj",
          "copy_from": "responseBody.results",
          "ignore_empty_value": true
        }
      },
      {
        "set": {
          "field": "embedded_error_code",
          "copy_from": "embedded_original_value_obj.statusCode",
          "ignore_empty_value": true
        }
      }
    ]
  }
}

{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "embedded_original_value_obj" : {
            "statusCode" : "200"
          },
          "responseBody" : {
            "results" : {
              "statusCode" : "200"
            }
          },
          "embedded_error_code" : "200"
        },
        "_ingest" : {
          "timestamp" : "2022-03-01T16:15:52.708791852Z"
        }
      }
    }
  ]
}

"2. set" looks also working in my environment. What you want to do?

In addition, you have to remove unnecessary fields (or set dynamic paramter to false) to avoid mapping_parsing_exception. I suppose rename processor is better for you if there is no other reason.

[shivani_singh]

@Tomo_M

ition, you have to remove unn

Hi @Tomo_M ,
Thank you for your efforts. I resolved this issue by using for each processor in the place of 2nd set, one of the community members helped me through this -
Set processor isn't working after another Set processor - Injest Pipeline - Elastic Stack / Elasticsearch - Discuss the Elastic Stack

Thank you.