# Ingest pipeline error when accessing a field

**URL:** <https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847>\
**Category:** Elasticsearch\
**Tags:** ingest-pipeline\
**Created:** [March 1, 2021, 6:58pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847 "2021-03-01T18:58:02Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![maheshe](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maheshe](https://discuss.elastic.co/u/maheshe)\
**Post date:** [March 1, 2021, 6:58pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/1 "2021-03-01T18:58:02Z")

</div>

I get this error message for the last Renameprocessor (Refer the last rule in Ingest Template). Iam running the Testpipeline. What is the right way to access the targetResources[userPrincipalName] in json.

```auto
 "error": { "message": "field [targetResources.userPrincipalName] doesn't exist" ]}

```

This is the Ingest template

```auto
[
  {
    "date": {
      "field": "activityDateTime",
      "formats": [
        "ISO8601"
      ],
      "description": "set @timestamp"
    }
  },
  {
    "rename": {
      "field": "activityDisplayName",
      "target_field": "event.action",
      "description": "activityDisplayName -> event.action"
    }
  },
  {
    "rename": {
      "field": "initiatedBy.user.userPrincipalName",
      "target_field": "IntiatedByUser",
      "description": "initiatedBy.userPrincipalName -> IntiatedByUser"
    }
  },
  {
    "rename": {
      "field": "targetResources.userPrincipalName",
      "target_field": "TargetToUser",
      "description": "targetResources.userPrincipalName -> TargetToUser"
    }
  }
]

```

This is the input json am using to test

```auto
{
  "_index": "event.audit-group-api_data-2021-02-25-000001",
  "_id": "WKdD33cBtqMuzOE8PK3s",
   "_source": {
    "id": "Directory_0e4fdfae-ad82-49a9-968b-e3f1dff55526_5HP78_48797866",
    "category": "GroupManagement",
    "correlationId": "0e4fdfae-ad82-49a9-968b-e3f1dff55526",
    "result": "success",
    "resultReason": "",
    "activityDisplayName": "Remove member from group",
    "activityDateTime": "2021-02-22T22:22:26.0666217Z",
    "loggedByService": "Core Directory",
    "operationType": "Unassign",
    "initiatedBy": {
      "app": null,
      "user": {
        "id": "saasasa-1457-4017-856a-39f81800aa10",
        "displayName": null,
        "userPrincipalName": "admin@mailerdev.onmicrosoft.com",
        "ipAddress": null
      }
    },
    "targetResources": [
      {
        "id": "7c7a1441-494c-43a2-bd83-88a49297eefb",
        "displayName": null,
        "type": "User",
        "userPrincipalName": "userwhoneedsgroupchange@mailerdev.onmicrosoft.com",
        "groupType": null,
        "modifiedProperties": [
          {
            "displayName": "Group.ObjectID",
            "oldValue": null,
            "newValue": "\"2aec66af-384a-43ba-86f6-73cb4162b884\""
          },
          {
            "displayName": "Group.DisplayName",
            "oldValue": null,
            "newValue": "\"Audit Log Test\""
          },
          {
            "displayName": "Group.WellKnownObjectName",
            "oldValue": null,
            "newValue": null
          }
        ]
      },
      {
        "id": "2aec66af-384a-43ba-86f6-73cb4162b884",
        "displayName": null,
        "type": "Group",
        "userPrincipalName": null,
        "groupType": "unknownFutureValue",
        "modifiedProperties": []
      }
    ],
    "additionalDetails": [],
    "REMOTE_ADDR": "110.129.11.99",
    "@timestamp": "2021-02-26T16:54:15.860831019+00:00"
  }
}

```

---

<div class="post-metadata">

**Author:** ![maheshe](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maheshe](https://discuss.elastic.co/u/maheshe)\
**Post date:** [March 5, 2021, 4:58am UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/2 "2021-03-05T04:58:45Z")

</div>

I modified the ingest template and tried to use the JSON processor to traverse & retrieve the targetResources[userPrincipalName] but unable to do so  
_Reference:[JSON processor | Elasticsearch Reference [7.x] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/json-processor.html)_

```auto
{
  "field": "targetResources[userPrincipalName]",
  "add_to_root": true
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 5, 2021, 6:05am UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/3 "2021-03-05T06:05:09Z")

</div>

Hi @maheshe

The problem is that `targetResources` is an Array of Objects not just a list of subfields, see the `[]`. I am not sure if you intend that to be the case or not, but that implies there could be more than 1 `userPrincipalName` sub-object / field. That is why rename won't work.

This will complicate matters a bit.

You have a couple choices if you process it as an array will will possibly need more than 1 place to put the `userPrincipalName` or you can flatten the data if you have control over that.

If you are always sure there is just 1 object in the targetResources array you could try using the [foreach](https://www.elastic.co/guide/en/elasticsearch/reference/7.x/foreach-processor.html) processor, or you could make an array of `TargetToUser`

Or you could access the first element in the array as seen below, but if the array has more than 1 element you are not guaranteed which one you will get.

```
PUT _ingest/pipeline/test
{
  "description": "",
  "processors": [
  {
    "date": {
      "field": "activityDateTime",
      "formats": [
        "ISO8601"
      ],
      "description": "set @timestamp"
    }
  },
  {
    "rename": {
      "field": "activityDisplayName",
      "target_field": "event.action",
      "description": "activityDisplayName -> event.action"
    }
  },
  {
    "rename": {
      "field": "initiatedBy.user.userPrincipalName",
      "target_field": "IntiatedByUser",
      "description": "initiatedBy.userPrincipalName -> IntiatedByUser"
    }
  },
  {
    "rename": {
      "field": "targetResources.0.userPrincipalName",
      "target_field": "TargetToUser",
      "description": "targetResources.userPrincipalName -> TargetToUser"
    }
  }
]
}

POST test-index/_doc/?pipeline=test

POST /_ingest/pipeline/test/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "id": "Directory_0e4fdfae-ad82-49a9-968b-e3f1dff55526_5HP78_48797866",
        "category": "GroupManagement",
        "correlationId": "0e4fdfae-ad82-49a9-968b-e3f1dff55526",
        "result": "success",
        "resultReason": "",
        "activityDisplayName": "Remove member from group",
        "activityDateTime": "2021-02-22T22:22:26.0666217Z",
        "loggedByService": "Core Directory",
        "operationType": "Unassign",
        "initiatedBy": {
          "app": null,
          "user": {
            "id": "saasasa-1457-4017-856a-39f81800aa10",
            "displayName": null,
            "userPrincipalName": "admin@mailerdev.onmicrosoft.com",
            "ipAddress": null
          }
        },
        "targetResources": [
          {
            "id": "7c7a1441-494c-43a2-bd83-88a49297eefb",
            "displayName": null,
            "type": "User",
            "userPrincipalName": "userwhoneedsgroupchange@mailerdev.onmicrosoft.com",
            "groupType": null,
            "modifiedProperties": [
              {
                "displayName": "Group.ObjectID",
                "oldValue": null,
                "newValue": "\"2aec66af-384a-43ba-86f6-73cb4162b884\""
              },
              {
                "displayName": "Group.DisplayName",
                "oldValue": null,
                "newValue": "\"Audit Log Test\""
              },
              {
                "displayName": "Group.WellKnownObjectName",
                "oldValue": null,
                "newValue": null
              }
            ]
          },
          {
            "id": "2aec66af-384a-43ba-86f6-73cb4162b884",
            "displayName": null,
            "type": "Group",
            "userPrincipalName": null,
            "groupType": "unknownFutureValue",
            "modifiedProperties": []
          }
        ],
        "additionalDetails": [],
        "REMOTE_ADDR": "110.129.11.99",
        "@timestamp": "2021-02-26T16:54:15.860831019+00:00"
      }
    }
  ]
}

# Result

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "TargetToUser" : "userwhoneedsgroupchange@mailerdev.onmicrosoft.com", <---- Renamed Here
          "REMOTE_ADDR" : "110.129.11.99",
          "IntiatedByUser" : "admin@mailerdev.onmicrosoft.com",
          "activityDateTime" : "2021-02-22T22:22:26.0666217Z",
          "additionalDetails" : [],
          "result" : "success",
          "@timestamp" : "2021-02-22T22:22:26.066Z",
          "resultReason" : "",
          "correlationId" : "0e4fdfae-ad82-49a9-968b-e3f1dff55526",
          "loggedByService" : "Core Directory",
          "operationType" : "Unassign",
          "id" : "Directory_0e4fdfae-ad82-49a9-968b-e3f1dff55526_5HP78_48797866",
          "category" : "GroupManagement",
          "targetResources" : [
            {
              "modifiedProperties" : [
                {
                  "newValue" : "\"2aec66af-384a-43ba-86f6-73cb4162b884\"",
                  "displayName" : "Group.ObjectID",
                  "oldValue" : null
                },
                {
                  "newValue" : "\"Audit Log Test\"",
                  "displayName" : "Group.DisplayName",
                  "oldValue" : null
                },
                {
                  "newValue" : null,
                  "displayName" : "Group.WellKnownObjectName",
                  "oldValue" : null
                }
              ],
              "groupType" : null,
              "id" : "7c7a1441-494c-43a2-bd83-88a49297eefb",
              "type" : "User",
              "displayName" : null
            },
            {
              "modifiedProperties" : [],
              "groupType" : "unknownFutureValue",
              "id" : "2aec66af-384a-43ba-86f6-73cb4162b884",
              "type" : "Group",
              "displayName" : null,
              "userPrincipalName" : null
            }
          ],
          "event" : {
            "action" : "Remove member from group"
          },
          "initiatedBy" : {
            "app" : null,
            "user" : {
              "ipAddress" : null,
              "displayName" : null,
              "id" : "saasasa-1457-4017-856a-39f81800aa10"
            }
          }
        },
        "_ingest" : {
          "timestamp" : "2021-03-05T06:01:43.553641102Z"
        }
      }
    }
  ]
}
```

---

<div class="post-metadata">

**Author:** ![maheshe](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maheshe](https://discuss.elastic.co/u/maheshe)\
**Post date:** [March 5, 2021, 10:41pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/4 "2021-03-05T22:41:00Z")

</div>

Thanks @stephenb this information helps.  
I don't have control over the input json. Best way as you suggested is to put a foreach processor and pull the record.

---

<div class="post-metadata">

**Author:** ![maheshe](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maheshe](https://discuss.elastic.co/u/maheshe)\
**Post date:** [March 16, 2021, 4:58pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/5 "2021-03-16T16:58:46Z")

</div>

Hi Stephen,  
Iam trying to run this Ingest Foreach processor to iterate the modified properties, I get a compile time exception when I try to run it. what am I doing wrong here

```auto
{
  "field": "targetResources.0.modifiedProperties",
  "processor": {
    "set": {
      "if": "_ingest._value.displayName.contains('DisplayName')",
      "field": "_ingest._value.newValue",
      "value": "ChangedField"
    }
  }
}

```

Thanks

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 16, 2021, 5:06pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/6 "2021-03-16T17:06:13Z")

</div>

That does not look like the `foreach` processor or it is not the complete please post the entire `foreach` block

What is the exception?

What is your desired output look like?

Did you looks at [the examples](https://www.elastic.co/guide/en/elasticsearch/reference/7.11/foreach-processor.html)?

---

<div class="post-metadata">

**Author:** ![maheshe](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maheshe](https://discuss.elastic.co/u/maheshe)\
**Post date:** [March 16, 2021, 5:52pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/7 "2021-03-16T17:52:02Z")

</div>

.. I have logged onto Kibana and navigated to Kibana-\>Ingest NodePipeline in the console and editing the IngestTemplate with a new processor- foreach processor. I have used the above input JSON. It doesn't save the pipeline but fails.

Result expected:

- I want to iterate through modified properties
- If the value of displayName is containing "Name"
- Want to add a new field in the root with the newName value

```auto
The array which gets iterated through 
{
              "displayName": "Group.DisplayName",
              "oldValue": null,
              "newValue": "\"Audit Log Test\""
            }

```

This is the error when I run the TestPipeline

```auto
Unable to execute pipeline.
[script_exception] compile error, with { processor_type="foreach" & script_stack={ 0="_ingest._value.displayNam ..." & 1="^---- HERE" } & script="_ingest._value.displayName.contains('DisplayName')" & lang="painless" & position={ offset=0 & start=0 & end=25 } }

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 16, 2021, 6:14pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/8 "2021-03-16T18:14:22Z")

</div>

What if there is more than 1 field with `displayName` that contains "Name" what is that to look like at the root level fields.

Plus you have and Array within and Array I am not sure how to handle that, I am not sure if you can nest a foreach within a foreach and I am not and expert iterations with painless.

---

<div class="post-metadata">

**Author:** ![maheshe](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maheshe](https://discuss.elastic.co/u/maheshe)\
**Post date:** [March 16, 2021, 9:06pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/9 "2021-03-16T21:06:52Z")

</div>

This for each works and converts the attribute to uppercase, I want to add an If statement to check the value of displayName and then if displayname contains "name" to be able to extract newName...  
Atleast one of the entries will have displayname as per the API specs so there wont be more than 1 displayname

```auto
{
  "field": "targetResources.0.modifiedProperties",
  "processor": {
    "uppercase": {
      "field": "_ingest._value.displayName"
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 16, 2021, 9:41pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/10 "2021-03-16T21:41:55Z")

</div>

Hi @maheshe can you post the whole foreach processor that you are building (show above) that looks like just the inside, The more you give use the more we can help?

---

<div class="post-metadata">

**Author:** ![maheshe](https://avatars.discourse-cdn.com/v4/letter/m/b19c9b/32.png) [@maheshe](https://discuss.elastic.co/u/maheshe)\
**Post date:** [March 16, 2021, 10:06pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/11 "2021-03-16T22:06:12Z")

</div>

This is the ingest Template the last processor is the one Iam trying to script

```auto
[
  {
    "date": {
      "field": "activityDateTime",
      "formats": [
        "ISO8601"
      ],
      "target_field": "sampleDate",
      "description": "settingdates"
    }
  },
  {
    "rename": {
      "field": "activityDisplayName",
      "target_field": "event.action",
      "description": "activityDisplayName -> event.action"
    }
  },
  {
    "rename": {
      "field": "initiatedBy.user.userPrincipalName",
      "target_field": "IntiatedByUser",
      "description": "initiatedBy.userPrincipalName -> IntiatedByUser"
    }
  },
  {
    "rename": {
      "field": "targetResources.0.userPrincipalName",
      "target_field": "TargetToUser",
      "description": "targetResources.userPrincipalName -> TargetToUser"
    }
  },
  {
    "foreach": {
      "field": "targetResources.0.modifiedProperties",
      "processor": {
        "uppercase": {
          "field": "_ingest._value.displayName"
        }
      }
    }
  }
]

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 13, 2021, 10:06pm UTC](https://discuss.elastic.co/t/ingest-pipeline-error-when-accessing-a-field/265847/12 "2021-04-13T22:06:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.