# Set processor isn't working after another Set processor - Injest Pipeline

**URL:** <https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [March 1, 2022, 7:40am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478 "2022-03-01T07:40:51Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [March 1, 2022, 7:40am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/1 "2022-03-01T07:40:51Z")

</div>

Hi,

I'm copying value of `responseBody.results` into `embedded_original_value_obj` both are  
`object` type fields.

> This is my defined pipeline -

```auto
PUT _ingest/pipeline/apiproxy-embedded-copy
{
  "processors": [
    {
   "set": {
        "field": "embedded_original_value_obj",
        "copy_from": "responseBody.results",
        "ignore_empty_value": true
      }
    },
    {
   "set": {
        "field": "embedded_error_code",
        "copy_from": "embedded_original_value_obj.statusCode",
        "ignore_empty_value": true
      }
    }
]
}

```

> For Testing I'm using this data -

```auto
POST _ingest/pipeline/apiproxy-embedded-copy/_simulate
{
  "docs": [
    {
      "_source": {
        "responseBody": {
          "results": [
            {
              "userMessage": "SUCCESS",
              "hasErrors": false,
              "developerMessage": "SUCCESS",
              "customerId": "1420371438",
              "messageId": null,
              "uniqueId": "P0000049373",
              "errors": [],
              "statusCode": 201
            }
          ]
        }
      }
    }
  ]
}

```

> output

```auto
{
  "docs" : [
    {
      "doc" : {
        "_index" : "_index",
        "_type" : "_doc",
        "_id" : "_id",
        "_source" : {
          "embedded_original_value_obj" : [
            {
              "userMessage" : "SUCCESS",
              "hasErrors" : false,
              "developerMessage" : "SUCCESS",
              "customerId" : "1420371438",
              "messageId" : null,
              "uniqueId" : "P0000049373",
              "errors" : [],
              "statusCode" : 201
            }
          ],
          "responseBody" : {
            "results" : [
              {
                "userMessage" : "SUCCESS",
                "hasErrors" : false,
                "developerMessage" : "SUCCESS",
                "customerId" : "1420371438",
                "messageId" : null,
                "uniqueId" : "P0000049373",
                "errors" : [],
                "statusCode" : 201
              }
            ]
          }
        },
        "_ingest" : {
          "timestamp" : "2022-03-01T07:35:59.914902863Z"
        }
      }
    }
  ]
}

```

`1st set` is working and I'm getting value in `embedded_original_value_obj` but `2nd set` isn't working. Any idea how can get value of `embedded_original_value_obj.statusCode` value into `embedded_error_code` ❓

---

<div class="post-metadata">

**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\
**Post date:** [March 1, 2022, 8:14am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/2 "2022-03-01T08:14:28Z")

</div>

Are you sure responseBody.results is an array, if not, you can do as follows

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "embedded_original_value_obj",
          "copy_from": "responseBody.results",
          "ignore_empty_value": true
        }
      },
      {
        "set": {
          "field": "embedded_error_code",
          "copy_from": "embedded_original_value_obj.statusCode",
          "ignore_empty_value": true
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "responseBody": {
          "results": {
            "userMessage": "SUCCESS",
            "hasErrors": false,
            "developerMessage": "SUCCESS",
            "customerId": "1420371438",
            "messageId": null,
            "uniqueId": "P0000049373",
            "errors": [],
            "statusCode": 201
          }
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [March 1, 2022, 9:16am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/3 "2022-03-01T09:16:58Z")

</div>

Hi @casterQ ,  
This could be working fine for me (and it did work with your above data ) if `results` was a `json` object but it is actually a `list`. The `responseBody` data comes as -

```auto
"responseBody" : {
            "results" : [
              {
                "statusCode" : 201
              }
            ]
          }

```

---

<div class="post-metadata">

**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\
**Post date:** [March 1, 2022, 9:23am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/4 "2022-03-01T09:23:06Z")

</div>

if it is a list(or array), so there will be many statusCode here，so you want embedded\_error\_code field to be as shown below？

```auto
"embedded_error_code":[201,202,203]

```

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [March 1, 2022, 9:29am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/5 "2022-03-01T09:29:31Z")

</div>

There will be only 1 `statuscode` per document i.e. `results[]` only contain 1 `statusCode` within a `responseBody`.

I want `embedded_error_code` to be created for every doc where `embedded_original_value_obj` got created.

so structure should be -  
`"embedded_error_code" : 201`

---

<div class="post-metadata">

**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\
**Post date:** [March 1, 2022, 9:50am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/6 "2022-03-01T09:50:43Z")

</div>

Does this meet your needs？

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "embedded_original_value_obj",
          "copy_from": "responseBody.results",
          "ignore_empty_value": true
        }
      },
      {
        "foreach": {
          "field": "embedded_original_value_obj",
          "processor": {
            "set": {
              "field": "embedded_error_code",
              "if":"ctx.embedded_error_code==null",
              "value": "{{_ingest._value.statusCode}}"
            }
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "responseBody": {
          "results": [
            {
              "userMessage": "SUCCESS",
              "hasErrors": false,
              "developerMessage": "SUCCESS",
              "customerId": "1420371438",
              "messageId": null,
              "uniqueId": "P0000049373",
              "errors": [],
              "statusCode": 201
            },
            {
              "userMessage": "SUCCESS",
              "hasErrors": false,
              "developerMessage": "SUCCESS",
              "customerId": "1420371438",
              "messageId": null,
              "uniqueId": "P0000049373",
              "errors": []
            }
          ]
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\
**Post date:** [March 1, 2022, 9:53am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/7 "2022-03-01T09:53:36Z")

</div>

is it ok？

---

<div class="post-metadata">

**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\
**Post date:** [March 1, 2022, 10:00am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/8 "2022-03-01T10:00:49Z")

</div>

rename statusCode to embedded\_error\_code ：

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "embedded_original_value_obj",
          "copy_from": "responseBody.results",
          "ignore_empty_value": true
        }
      },
      {
        "foreach": {
          "field": "embedded_original_value_obj",
          "processor": {
            "rename": {
              "field": "_ingest._value.statusCode",
              "target_field": "_ingest._value.embedded_error_code"
            }
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "responseBody": {
          "results": [
            {
              "userMessage": "SUCCESS",
              "hasErrors": false,
              "developerMessage": "SUCCESS",
              "customerId": "1420371438",
              "messageId": null,
              "uniqueId": "P0000049373",
              "errors": [],
              "statusCode": 201
            },
            {
              "userMessage": "SUCCESS",
              "hasErrors": false,
              "developerMessage": "SUCCESS",
              "customerId": "1420371438",
              "messageId": null,
              "uniqueId": "P0000049373",
              "errors": [],
              "statusCode": 202
            }
          ]
        }
      }
    }
  ]
}

```

Add additional field embedded\_error\_code:

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "embedded_original_value_obj",
          "copy_from": "responseBody.results",
          "ignore_empty_value": true
        }
      },
      {
        "foreach": {
          "field": "embedded_original_value_obj",
          "processor": {
            "set": {
              "field": "_ingest._value.embedded_error_code",
              "copy_from": "_ingest._value.statusCode"
            }
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "responseBody": {
          "results": [
            {
              "userMessage": "SUCCESS",
              "hasErrors": false,
              "developerMessage": "SUCCESS",
              "customerId": "1420371438",
              "messageId": null,
              "uniqueId": "P0000049373",
              "errors": [],
              "statusCode": 201
            },
            {
              "userMessage": "SUCCESS",
              "hasErrors": false,
              "developerMessage": "SUCCESS",
              "customerId": "1420371438",
              "messageId": null,
              "uniqueId": "P0000049373",
              "errors": [],
              "statusCode": 202
            }
          ]
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [March 1, 2022, 10:10am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/9 "2022-03-01T10:10:32Z")

</div>

Hi @casterQ ,  
It worked with `_simulate` (above code you mentioned) but it's giving me an error while I reindexed my data.

```auto
"cause" : {
        "type" : "illegal_argument_exception",
        "reason" : "field [embedded_original_value_obj] not present as part of path [embedded_original_value_obj]"
      },

```

---

<div class="post-metadata">

**Author:** ![casterQ](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/casterq/32/93257_2.png) [@casterQ](https://discuss.elastic.co/u/casterQ)\
**Post date:** [March 1, 2022, 10:16am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/11 "2022-03-01T10:16:03Z")

</div>

Which of the above three DSL is you need?

---

<div class="post-metadata">

**Author:** ![shivani\_singh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shivani_singh/32/102450_2.png) [@shivani\_singh](https://discuss.elastic.co/u/shivani_singh)\
**Post date:** [March 1, 2022, 10:22am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/12 "2022-03-01T10:22:11Z")

</div>

Hi @casterQ ,  
1st pipeline you gave is working for now!!!

```auto
POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
        "set": {
          "field": "embedded_original_value_obj",
          "copy_from": "responseBody.results",
          "ignore_empty_value": true
        }
      },
      {
        "foreach": {
          "field": "embedded_original_value_obj",
          "processor": {
            "set": {
              "field": "embedded_error_code",
              "if":"ctx.embedded_error_code==null",
              "value": "{{_ingest._value.statusCode}}"
            }
          }
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
        "responseBody": {
          "results": [
            {
              "userMessage": "SUCCESS",
              "hasErrors": false,
              "developerMessage": "SUCCESS",
              "customerId": "1420371438",
              "messageId": null,
              "uniqueId": "P0000049373",
              "errors": [],
              "statusCode": 201
            }
          ]
        }
      }
    }
  ]
}

```

Thank you so much for quick response, really appreciate your efforts. 🙏

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 29, 2022, 10:22am UTC](https://discuss.elastic.co/t/set-processor-isnt-working-after-another-set-processor-injest-pipeline/298478/13 "2022-03-29T10:22:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.