Ingest pipeline error when accessing a field

Elastic Stack Elasticsearch
1

I get this error message for the last Renameprocessor (Refer the last rule in Ingest Template). Iam running the Testpipeline. What is the right way to access the targetResources[userPrincipalName] in json.

 "error": { "message": "field [targetResources.userPrincipalName] doesn't exist" ]}

This is the Ingest template

[
  {
    "date": {
      "field": "activityDateTime",
      "formats": [
        "ISO8601"
      ],
      "description": "set @timestamp"
    }
  },
  {
    "rename": {
      "field": "activityDisplayName",
      "target_field": "event.action",
      "description": "activityDisplayName -> event.action"
    }
  },
  {
    "rename": {
      "field": "initiatedBy.user.userPrincipalName",
      "target_field": "IntiatedByUser",
      "description": "initiatedBy.userPrincipalName -> IntiatedByUser"
    }
  },
  {
    "rename": {
      "field": "targetResources.userPrincipalName",
      "target_field": "TargetToUser",
      "description": "targetResources.userPrincipalName -> TargetToUser"
    }
  }
]

This is the input json am using to test

{
  "_index": "event.audit-group-api_data-2021-02-25-000001",
  "_id": "WKdD33cBtqMuzOE8PK3s",
   "_source": {
    "id": "Directory_0e4fdfae-ad82-49a9-968b-e3f1dff55526_5HP78_48797866",
    "category": "GroupManagement",
    "correlationId": "0e4fdfae-ad82-49a9-968b-e3f1dff55526",
    "result": "success",
    "resultReason": "",
    "activityDisplayName": "Remove member from group",
    "activityDateTime": "2021-02-22T22:22:26.0666217Z",
    "loggedByService": "Core Directory",
    "operationType": "Unassign",
    "initiatedBy": {
      "app": null,
      "user": {
        "id": "saasasa-1457-4017-856a-39f81800aa10",
        "displayName": null,
        "userPrincipalName": "admin@mailerdev.onmicrosoft.com",
        "ipAddress": null
      }
    },
    "targetResources": [
      {
        "id": "7c7a1441-494c-43a2-bd83-88a49297eefb",
        "displayName": null,
        "type": "User",
        "userPrincipalName": "userwhoneedsgroupchange@mailerdev.onmicrosoft.com",
        "groupType": null,
        "modifiedProperties": [
          {
            "displayName": "Group.ObjectID",
            "oldValue": null,
            "newValue": "\"2aec66af-384a-43ba-86f6-73cb4162b884\""
          },
          {
            "displayName": "Group.DisplayName",
            "oldValue": null,
            "newValue": "\"Audit Log Test\""
          },
          {
            "displayName": "Group.WellKnownObjectName",
            "oldValue": null,
            "newValue": null
          }
        ]
      },
      {
        "id": "2aec66af-384a-43ba-86f6-73cb4162b884",
        "displayName": null,
        "type": "Group",
        "userPrincipalName": null,
        "groupType": "unknownFutureValue",
        "modifiedProperties": []
      }
    ],
    "additionalDetails": [],
    "REMOTE_ADDR": "110.129.11.99",
    "@timestamp": "2021-02-26T16:54:15.860831019+00:00"
  }
}
2

I modified the ingest template and tried to use the JSON processor to traverse & retrieve the targetResources[userPrincipalName] but unable to do so
Reference:JSON processor | Elasticsearch Reference [7.x] | Elastic

{
  "field": "targetResources[userPrincipalName]",
  "add_to_root": true
}
3

Hi @maheshe

The problem is that targetResources is an Array of Objects not just a list of subfields, see the []. I am not sure if you intend that to be the case or not, but that implies there could be more than 1 userPrincipalName sub-object / field. That is why rename won't work.

This will complicate matters a bit.

You have a couple choices if you process it as an array will will possibly need more than 1 place to put the userPrincipalName or you can flatten the data if you have control over that.

If you are always sure there is just 1 object in the targetResources array you could try using the foreach processor, or you could make an array of TargetToUser

Or you could access the first element in the array as seen below, but if the array has more than 1 element you are not guaranteed which one you will get.

PUT _ingest/pipeline/test
{
  "description": "",
  "processors": [
  {
    "date": {
      "field": "activityDateTime",
      "formats": [
        "ISO8601"
      ],
      "description": "set @timestamp"
    }
  },
  {
    "rename": {
      "field": "activityDisplayName",
      "target_field": "event.action",
      "description": "activityDisplayName -> event.action"
    }
  },
  {
    "rename": {
      "field": "initiatedBy.user.userPrincipalName",
      "target_field": "IntiatedByUser",
      "description": "initiatedBy.userPrincipalName -> IntiatedByUser"
    }
  },
  {
    "rename": {
      "field": "targetResources.0.userPrincipalName",
      "target_field": "TargetToUser",
      "description": "targetResources.userPrincipalName -> TargetToUser"
    }
  }
]
}

POST test-index/_doc/?pipeline=test

POST /_ingest/pipeline/test/_simulate
{
  "docs": [
    {
      "_index": "index",
      "_id": "id",
      "_source": {
        "id": "Directory_0e4fdfae-ad82-49a9-968b-e3f1dff55526_5HP78_48797866",
        "category": "GroupManagement",
        "correlationId": "0e4fdfae-ad82-49a9-968b-e3f1dff55526",
        "result": "success",
        "resultReason": "",
        "activityDisplayName": "Remove member from group",
        "activityDateTime": "2021-02-22T22:22:26.0666217Z",
        "loggedByService": "Core Directory",
        "operationType": "Unassign",
        "initiatedBy": {
          "app": null,
          "user": {
            "id": "saasasa-1457-4017-856a-39f81800aa10",
            "displayName": null,
            "userPrincipalName": "admin@mailerdev.onmicrosoft.com",
            "ipAddress": null
          }
        },
        "targetResources": [
          {
            "id": "7c7a1441-494c-43a2-bd83-88a49297eefb",
            "displayName": null,
            "type": "User",
            "userPrincipalName": "userwhoneedsgroupchange@mailerdev.onmicrosoft.com",
            "groupType": null,
            "modifiedProperties": [
              {
                "displayName": "Group.ObjectID",
                "oldValue": null,
                "newValue": "\"2aec66af-384a-43ba-86f6-73cb4162b884\""
              },
              {
                "displayName": "Group.DisplayName",
                "oldValue": null,
                "newValue": "\"Audit Log Test\""
              },
              {
                "displayName": "Group.WellKnownObjectName",
                "oldValue": null,
                "newValue": null
              }
            ]
          },
          {
            "id": "2aec66af-384a-43ba-86f6-73cb4162b884",
            "displayName": null,
            "type": "Group",
            "userPrincipalName": null,
            "groupType": "unknownFutureValue",
            "modifiedProperties": []
          }
        ],
        "additionalDetails": [],
        "REMOTE_ADDR": "110.129.11.99",
        "@timestamp": "2021-02-26T16:54:15.860831019+00:00"
      }
    }
  ]
}

# Result

{
  "docs" : [
    {
      "doc" : {
        "_index" : "index",
        "_type" : "_doc",
        "_id" : "id",
        "_source" : {
          "TargetToUser" : "userwhoneedsgroupchange@mailerdev.onmicrosoft.com", <---- Renamed Here
          "REMOTE_ADDR" : "110.129.11.99",
          "IntiatedByUser" : "admin@mailerdev.onmicrosoft.com",
          "activityDateTime" : "2021-02-22T22:22:26.0666217Z",
          "additionalDetails" : [ ],
          "result" : "success",
          "@timestamp" : "2021-02-22T22:22:26.066Z",
          "resultReason" : "",
          "correlationId" : "0e4fdfae-ad82-49a9-968b-e3f1dff55526",
          "loggedByService" : "Core Directory",
          "operationType" : "Unassign",
          "id" : "Directory_0e4fdfae-ad82-49a9-968b-e3f1dff55526_5HP78_48797866",
          "category" : "GroupManagement",
          "targetResources" : [
            {
              "modifiedProperties" : [
                {
                  "newValue" : "\"2aec66af-384a-43ba-86f6-73cb4162b884\"",
                  "displayName" : "Group.ObjectID",
                  "oldValue" : null
                },
                {
                  "newValue" : "\"Audit Log Test\"",
                  "displayName" : "Group.DisplayName",
                  "oldValue" : null
                },
                {
                  "newValue" : null,
                  "displayName" : "Group.WellKnownObjectName",
                  "oldValue" : null
                }
              ],
              "groupType" : null,
              "id" : "7c7a1441-494c-43a2-bd83-88a49297eefb",
              "type" : "User",
              "displayName" : null
            },
            {
              "modifiedProperties" : [ ],
              "groupType" : "unknownFutureValue",
              "id" : "2aec66af-384a-43ba-86f6-73cb4162b884",
              "type" : "Group",
              "displayName" : null,
              "userPrincipalName" : null
            }
          ],
          "event" : {
            "action" : "Remove member from group"
          },
          "initiatedBy" : {
            "app" : null,
            "user" : {
              "ipAddress" : null,
              "displayName" : null,
              "id" : "saasasa-1457-4017-856a-39f81800aa10"
            }
          }
        },
        "_ingest" : {
          "timestamp" : "2021-03-05T06:01:43.553641102Z"
        }
      }
    }
  ]
}
4

Thanks @stephenb this information helps.
I don't have control over the input json. Best way as you suggested is to put a foreach processor and pull the record.

5

Hi Stephen,
Iam trying to run this Ingest Foreach processor to iterate the modified properties, I get a compile time exception when I try to run it. what am I doing wrong here

{
  "field": "targetResources.0.modifiedProperties",
  "processor": {
    "set": {
      "if": "_ingest._value.displayName.contains('DisplayName')",
      "field": "_ingest._value.newValue",
      "value": "ChangedField"
    }
  }
}

Thanks

6

That does not look like the foreach processor or it is not the complete please post the entire foreach block

What is the exception?

What is your desired output look like?

Did you looks at the examples?

7

.. I have logged onto Kibana and navigated to Kibana->Ingest NodePipeline in the console and editing the IngestTemplate with a new processor- foreach processor. I have used the above input JSON. It doesn't save the pipeline but fails.

Result expected:

  • I want to iterate through modified properties
  • If the value of displayName is containing "Name"
  • Want to add a new field in the root with the newName value
The array which gets iterated through 
{
              "displayName": "Group.DisplayName",
              "oldValue": null,
              "newValue": "\"Audit Log Test\""
            }

This is the error when I run the TestPipeline

Unable to execute pipeline.
[script_exception] compile error, with { processor_type="foreach" & script_stack={ 0="_ingest._value.displayNam ..." & 1="^---- HERE" } & script="_ingest._value.displayName.contains('DisplayName')" & lang="painless" & position={ offset=0 & start=0 & end=25 } }
8

What if there is more than 1 field with displayName that contains "Name" what is that to look like at the root level fields.

Plus you have and Array within and Array I am not sure how to handle that, I am not sure if you can nest a foreach within a foreach and I am not and expert iterations with painless.

9

This for each works and converts the attribute to uppercase, I want to add an If statement to check the value of displayName and then if displayname contains "name" to be able to extract newName...
Atleast one of the entries will have displayname as per the API specs so there wont be more than 1 displayname

{
  "field": "targetResources.0.modifiedProperties",
  "processor": {
    "uppercase": {
      "field": "_ingest._value.displayName"
    }
  }
}
10

Hi @maheshe can you post the whole foreach processor that you are building (show above) that looks like just the inside, The more you give use the more we can help?

11

This is the ingest Template the last processor is the one Iam trying to script

[
  {
    "date": {
      "field": "activityDateTime",
      "formats": [
        "ISO8601"
      ],
      "target_field": "sampleDate",
      "description": "settingdates"
    }
  },
  {
    "rename": {
      "field": "activityDisplayName",
      "target_field": "event.action",
      "description": "activityDisplayName -> event.action"
    }
  },
  {
    "rename": {
      "field": "initiatedBy.user.userPrincipalName",
      "target_field": "IntiatedByUser",
      "description": "initiatedBy.userPrincipalName -> IntiatedByUser"
    }
  },
  {
    "rename": {
      "field": "targetResources.0.userPrincipalName",
      "target_field": "TargetToUser",
      "description": "targetResources.userPrincipalName -> TargetToUser"
    }
  },
  {
    "foreach": {
      "field": "targetResources.0.modifiedProperties",
      "processor": {
        "uppercase": {
          "field": "_ingest._value.displayName"
        }
      }
    }
  }
]
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.