# Enriching Data with Custom Index

**URL:** <https://discuss.elastic.co/t/enriching-data-with-custom-index/364084>\
**Category:** Kibana\
**Created:** [July 30, 2024, 6:26pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084 "2024-07-30T18:26:17Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 6:26pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/1 "2024-07-30T18:26:17Z")

</div>

I'm trying to enrich an alert generating event with a user name. The alert generating event only currently contains the SID of the account and it's randomly generated. I created a custom index to contain the SID and account name from previous events that include both. But I can't figure out how to write these elements to the custom index. Support helped with the below snippet, but "index" is apparently not a supported processor type. Any other way to do this?

```auto
##Create the ingest pipline for the alert triggering events
PUT _ingest/pipeline/lenovo_account_creation
{
  "processors": [
    {
      "script": {
        "source": """
          if (ctx?.user?.target?.name != null && ctx.user.target.name.startsWith('lenovo_tmp_')) {
            Map accountMapping = ['sid': ctx.user.target.id, 'account_name': ctx.user.target.name];
            ctx.account_mapping = accountMapping;
          }
        """
      }
    },
    {
      "index": {
        "target_index": "lenovo_account_mappings",
        "if": "ctx.account_mapping != null"
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 30, 2024, 6:28pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/2 "2024-07-30T18:28:51Z")

</div>

Hi @FlyNavy Welcome to the community

What version are you on?

There is a standard approach for enriching data perhaps take a look at this

> **[Enrich your data | Elasticsearch Guide \[8.14\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-enriching-data.html)**

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 6:33pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/3 "2024-07-30T18:33:43Z")

</div>

Thanks. I do have the rest of the enrichment process in place by what I think is this process. I just can't get the data detected in the ingestion pipeline to write to the enriching index for the mapped pair.

```auto
PUT /_enrich/policy/lenovo_account_enrichment_policy
{
"match": {
 "indices": "lenovo_account_mappings",
 "match_field": "sid",
 "enrich_fields": ["account_name"]
}
}

POST /_enrich/policy/lenovo_account_enrichment_policy/_execute

PUT _ingest/pipeline/enrich_alert_events
{
"processors": [
 {
   "enrich": {
     "policy_name": "lenovo_account_enrichment_policy",
     "field": "user.target.id",
     "target_field": "related.user_info",
     "max_matches": "1"
   }
 },
 {
   "script": {
     "source": """
       if (ctx?.related?.user_info?.account_name != null && ctx.related.user_info.account_name.startsWith('lenovo_tmp_')) {
         ctx.event.ignore_alert = true;
       }
     """
   }
 }
]
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 30, 2024, 6:42pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/4 "2024-07-30T18:42:11Z")

</div>

You will need to show the source enrich data couple samples  
A couple of docs you want to enrich

The pipeline (which is what you are showing)  
I would probably take out the script part till you see the enrich work

You are not showing the results you can use the `_simulate` on the pipeline to see if it is working...

Give full samples and we can probably help (you can anonymize the data) just need to see it

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 30, 2024, 6:47pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/5 "2024-07-30T18:47:18Z")

</div>

Assuming you have created mappings for your indices?

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 6:51pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/6 "2024-07-30T18:51:02Z")

</div>

I've copied the first several lines of the event data that includes both name and random SID. Not sure the rest matters.

A user account was created. Subject: Security ID: S-1-5-18 Account Name: XXXXXXXX$ Account Domain: XXXXXXXXX Logon ID: 0x3E7 New Account: **Security ID: S-1-5-21-949799925-2528041633-2783471443-4440**  **Account Name: lenovo\_tmp\_sjstILTZ** Account Domain: XXXXXXX Attributes: SAM Account Name: lenovo\_tmp\_sjstILTZ Display Name: User Principal Name: - Home Directory: Home Drive: Script Path: Profile Path: User Workstations:

And here's the alert generating event.  
A member was added to a security-enabled local group. Subject: Security ID: S-1-5-18 Account Name: XXXXXXXX$ Account Domain: XXXXXXXXXX Logon ID: 0x3E7 Member: **Security ID: S-1-5-21-949799925-2528041633-2783471443-4440** Account Name: - Group: Security ID: S-1-5-32-544 Group Name: Administrators Group Domain: Builtin Additional Information: Privileges: -@timestampJul 25, 2024 @ 13:23:41.976agent.ephemeral\_idfd18652b-adeb-49b1-a2e2-97239f97df8aagent.hostnameSophosagent.id82295858-5992-45e2-bce1-966e78fb8d42agent.nameXXXXXXagent.typewinlogbeatagent.version8.0.0ecs.version1.12.0event.actionadded-member-to-groupevent.categoryiamevent.code4732event.created

Since the SID is random, I need to use the account name format "lenovo\_tmp\_\*\*\*\*\*\*\*\*' to track and stop alert generation for these events. I would like to enrich the second event listed with the account name so I can tell the rule to ignore these events.

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 6:53pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/7 "2024-07-30T18:53:20Z")

</div>

I have already created the custom index for the mapped pair values.

```auto
PUT /lenovo_account_mappings
{
"mappings": {
 "properties": {
   "sid": { "type": "keyword" },
   "account_name": { "type": "keyword" }
 }
}
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 30, 2024, 6:55pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/8 "2024-07-30T18:55:15Z")

</div>

Apologies I want to see what the documents look like in Elastic after you have ingested them not the raw logs ... the JSON representation of in elastic After you have ingested them.

Have you parsed the logs? What does that data look like?

If you have not parsed the logs into fields that will be your first step...

I see the mapping for the enrich data ... good...

I mean have you created a mapping for the logs, ingested and parsed them.

How are the logs getting ingested are you using filebeat, Elastic Agent?

What version are you on?

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 7:14pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/9 "2024-07-30T19:14:02Z")

</div>

Here's the message after ingestion that kicks off the alert. It contains only the SID

```auto
{
  "_index": ".ds-winlogbeat-8.0.0-2024.07.30-000570",
  "_id": "TMz7BJEBTE3soZyKwcni",
  "_version": 1,
  "_score": 0,
  "_source": {
    "agent": {
      "name": "XXXXX",
      "id": "82295858-5992-45e2-bce1-966e78fb8d42",
      "type": "winlogbeat",
      "ephemeral_id": "fd18652b-adeb-49b1-a2e2-97239f97df8a",
      "version": "8.0.0"
    },
    "@timestamp": "2024-07-30T18:50:59.567Z",
    "winlog": {
      "computer_name": "XXXX.XXXXX.XXXXXX",
      "process": {
        "pid": 672,
        "thread": {
          "id": 1096
        }
      },
      "keywords": [
        "Audit Success"
      ],
      "logon": {
        "id": "0x3e7"
      },
      "channel": "Security",
      "event_data": {
        "SubjectUserName": "XXXXXXX",
        "MemberSid": "S-1-5-21-2366788312-814031040-1301217162-1086",
        "TargetSid": "S-1-5-32-544",
        "SubjectDomainName": "XXXXXXXX",
        "SubjectLogonId": "0x3e7",
        "MemberName": "-",
        "TargetUserName": "Administrators",
        "TargetDomainName": "Builtin",
        "SubjectUserSid": "S-1-5-18",
        "PrivilegeList": "-"
      },
      "opcode": "Info",
      "record_id": "529303",
      "task": "Security Group Management",
      "event_id": "4732",
      "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
      "activity_id": "{41b78ff5-d8e1-0002-f890-b741e1d8da01}",
      "api": "wineventlog",
      "provider_name": "Microsoft-Windows-Security-Auditing"
    },
    "ecs": {
      "version": "1.12.0"
    },
    "related": {
      "user": [
        "-",
        "XXXXXXXX"
      ]
    },
    "log": {
      "level": "information"
    },
    "host": {
      "name": "XXXXXX.XXXXXX.XXX"
    },
    "event": {
      "ingested": "2024-07-30T18:51:33.724955676Z",
      "code": "4732",
      "provider": "Microsoft-Windows-Security-Auditing",
      "created": "2024-07-30T18:51:33.147Z",
      "kind": "event",
      "action": "added-member-to-group",
      "category": [
        "iam"
      ],
      "type": [
        "group",
        "change"
      ],
      "outcome": "success"
    },
    "message": "A member was added to a security-enabled local group.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tXXXXXXXXX\n\tAccount Domain:\t\tXXXXXXXXXX\n\tLogon ID:\t\t0x3E7\n\nMember:\n\tSecurity ID:\t\tS-1-5-21-2366788312-814031040-1301217162-1086\n\tAccount Name:\t\t-\n\nGroup:\n\tSecurity ID:\t\tS-1-5-32-544\n\tGroup Name:\t\tAdministrators\n\tGroup Domain:\t\tBuiltin\n\nAdditional Information:\n\tPrivileges:\t\t-",
    "user": {
      "domain": "XXXXXXXXXX",
      "name": "XXXXXXXX",
      "id": "S-1-5-18",
      "target": {
        "name": "-",
        "group": {
          "domain": "Builtin",
          "name": "Administrators",
          "id": "S-1-5-32-544"
        }
      }
    },
    "tags": [
      "forwarded"
    ],
    "group": {
      "domain": "Builtin",
      "name": "Administrators",
      "id": "S-1-5-32-544"
    }
  },
  "fields": {
    "event.category": [
      "iam"
    ],
    "winlog.provider_guid": [
      "{54849625-5478-4994-a5ba-3e3b0328c30d}"
    ],
    "winlog.provider_name": [
      "Microsoft-Windows-Security-Auditing"
    ],
    "user.target.group.domain": [
      "Builtin"
    ],
    "winlog.computer_name": [
      "XXXXXXX.XXXXXXXX.XXX"
    ],
    "winlog.process.pid": [
      672
    ],
    "winlog.keywords": [
      "Audit Success"
    ],
    "winlog.record_id": [
      "529303"
    ],
    "winlog.logon.id": [
      "0x3e7"
    ],
    "log.level": [
      "information"
    ],
    "agent.name": [
      "XXXXXXX"
    ],
    "host.name": [
      "XXXXX.XXXXXXXXXX.XXX"
    ],
    "user.target.name.text": [
      "-"
    ],
    "event.kind": [
      "event"
    ],
    "winlog.activity_id": [
      "{41b78ff5-d8e1-0002-f890-b741e1d8da01}"
    ],
    "event.outcome": [
      "success"
    ],
    "group.name": [
      "Administrators"
    ],
    "winlog.event_data.TargetUserName": [
      "Administrators"
    ],
    "user.id": [
      "S-1-5-18"
    ],
    "user.target.group.id": [
      "S-1-5-32-544"
    ],
    "agent.hostname": [
      "XXXXXXX"
    ],
    "related.user": [
      "-",
      "XXXXXXXXXX"
    ],
    "tags": [
      "forwarded"
    ],
    "user.target.name": [
      "-"
    ],
    "event.provider": [
      "Microsoft-Windows-Security-Auditing"
    ],
    "event.code": [
      "4732"
    ],
    "agent.id": [
      "82295858-5992-45e2-bce1-966e78fb8d42"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "event.created": [
      "2024-07-30T18:51:33.147Z"
    ],
    "agent.version": [
      "8.0.0"
    ],
    "winlog.event_data.SubjectUserSid": [
      "S-1-5-18"
    ],
    "winlog.process.thread.id": [
      1096
    ],
    "user.target.group.name": [
      "Administrators"
    ],
    "winlog.event_data.PrivilegeList": [
      "-"
    ],
    "group.id": [
      "S-1-5-32-544"
    ],
    "user.name": [
      "XXXXXXXXX"
    ],
    "agent.type": [
      "winlogbeat"
    ],
    "winlog.event_data.MemberName": [
      "-"
    ],
    "winlog.event_data.SubjectLogonId": [
      "0x3e7"
    ],
    "winlog.event_data.TargetSid": [
      "S-1-5-32-544"
    ],
    "group.domain": [
      "Builtin"
    ],
    "winlog.api": [
      "wineventlog"
    ],
    "user.domain": [
      "XXXXXXXXXXXX"
    ],
    "winlog.task": [
      "Security Group Management"
    ],
    "winlog.event_data.SubjectUserName": [
      "XXXXXXXXXX"
    ],
    "message": [
      "A member was added to a security-enabled local group.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tXXXXXXXXX\n\tAccount Domain:\t\tXXXXXXXXXXXXX\n\tLogon ID:\t\t0x3E7\n\nMember:\n\tSecurity ID:\t\tS-1-5-21-2366788312-814031040-1301217162-1086\n\tAccount Name:\t\t-\n\nGroup:\n\tSecurity ID:\t\tS-1-5-32-544\n\tGroup Name:\t\tAdministrators\n\tGroup Domain:\t\tBuiltin\n\nAdditional Information:\n\tPrivileges:\t\t-"
    ],
    "winlog.event_id": [
      "4732"
    ],
    "event.action": [
      "added-member-to-group"
    ],
    "event.ingested": [
      "2024-07-30T18:51:33.724Z"
    ],
    "@timestamp": [
      "2024-07-30T18:50:59.567Z"
    ],
    "winlog.channel": [
      "Security"
    ],
    "winlog.event_data.MemberSid": [
      "S-1-5-21-2366788312-814031040-1301217162-1086"
    ],
    "event.type": [
      "group",
      "change"
    ],
    "winlog.event_data.TargetDomainName": [
      "Builtin"
    ],
    "winlog.opcode": [
      "Info"
    ],
    "agent.ephemeral_id": [
      "fd18652b-adeb-49b1-a2e2-97239f97df8a"
    ],
    "winlog.event_data.SubjectDomainName": [
      "XXXXXXXXXX"
    ],
    "user.name.text": [
      "XXXXXXXXX"
    ]
  }
}

```

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 7:20pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/10 "2024-07-30T19:20:13Z")

</div>

And here's the account creation event

```auto
{
  "_index": ".ds-winlogbeat-8.0.0-2024.07.30-000570",
  "_id": "Fsz7BJEBTE3soZyKZrkQ",
  "_version": 1,
  "_score": 0,
  "_source": {
    "agent": {
      "name": "XXXXXXXX",
      "id": "82295858-5992-45e2-bce1-966e78fb8d42",
      "type": "winlogbeat",
      "ephemeral_id": "fd18652b-adeb-49b1-a2e2-97239f97df8a",
      "version": "8.0.0"
    },
    "@timestamp": "2024-07-30T18:50:59.547Z",
    "winlog": {
      "computer_name": "XXXXXX.XXXXXXXXXX.XXX",
      "process": {
        "pid": 672,
        "thread": {
          "id": 1096
        }
      },
      "keywords": [
        "Audit Success"
      ],
      "logon": {
        "id": "0x3e7"
      },
      "channel": "Security",
      "event_data": {
        "TargetSid": "S-1-5-21-2366788312-814031040-1301217162-1086",
        "NewUACList": [
          "LOCKOUT",
          "SCRIPT"
        ],
        "SidHistory": "-",
        "LogonHours": "%%1797",
        "ScriptPath": "%%1793",
        "DisplayName": "%%1793",
        "HomePath": "%%1793",
        "SubjectDomainName": "XXXXXXXXXX",
        "AllowedToDelegateTo": "-",
        "TargetDomainName": "XXXXXXXXX",
        "PrivilegeList": "-",
        "UserWorkstations": "%%1793",
        "SamAccountName": "lenovo_tmp_ygauCNFY",
        "UserParameters": "%%1793",
        "HomeDirectory": "%%1793",
        "OldUacValue": "0x0",
        "SubjectLogonId": "0x3e7",
        "NewUacValue": "0x15",
        "PrimaryGroupId": "513",
        "AccountExpires": "%%1794",
        "ProfilePath": "%%1793",
        "UserAccountControl": [
          "2080",
          "2082",
          "2084"
        ],
        "SubjectUserName": "XXXXXXXX",
        "PasswordLastSet": "%%1794",
        "UserPrincipalName": "-",
        "TargetUserName": "lenovo_tmp_ygauCNFY",
        "SubjectUserSid": "S-1-5-18"
      },
      "opcode": "Info",
      "record_id": "529299",
      "event_id": "4720",
      "task": "User Account Management",
      "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
      "activity_id": "{41b78ff5-d8e1-0002-f890-b741e1d8da01}",
      "api": "wineventlog",
      "provider_name": "Microsoft-Windows-Security-Auditing"
    },
    "ecs": {
      "version": "1.12.0"
    },
    "related": {
      "user": [
        "XXXXXXXXXXX",
        "lenovo_tmp_ygauCNFY"
      ]
    },
    "log": {
      "level": "information"
    },
    "host": {
      "name": "XXXXXXXX.XXXXXXXXXX.XXX"
    },
    "event": {
      "ingested": "2024-07-30T18:51:10.223631818Z",
      "code": "4720",
      "provider": "Microsoft-Windows-Security-Auditing",
      "created": "2024-07-30T18:51:10.110Z",
      "kind": "event",
      "action": "added-user-account",
      "type": [
        "user",
        "creation"
      ],
      "category": [
        "iam"
      ],
      "outcome": "success"
    },
    "message": "A user account was created.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tXXXXXXXXXXX\n\tAccount Domain:\t\tXXXXXXXXXXXX\n\tLogon ID:\t\t0x3E7\n\nNew Account:\n\tSecurity ID:\t\tS-1-5-21-2366788312-814031040-1301217162-1086\n\tAccount Name:\t\tlenovo_tmp_ygauCNFY\n\tAccount Domain:\t\tXXXXXXXXXX\n\nAttributes:\n\tSAM Account Name:\tlenovo_tmp_ygauCNFY\n\tDisplay Name:\t\t<value not set>\n\tUser Principal Name:\t-\n\tHome Directory:\t\t<value not set>\n\tHome Drive:\t\t<value not set>\n\tScript Path:\t\t<value not set>\n\tProfile Path:\t\t<value not set>\n\tUser Workstations:\t<value not set>\n\tPassword Last Set:\t<never>\n\tAccount Expires:\t\t<never>\n\tPrimary Group ID:\t513\n\tAllowed To Delegate To:\t-\n\tOld UAC Value:\t\t0x0\n\tNew UAC Value:\t\t0x15\n\tUser Account Control:\t\n\t\tAccount Disabled\n\t\t'Password Not Required' - Enabled\n\t\t'Normal Account' - Enabled\n\tUser Parameters:\t<value not set>\n\tSID History:\t\t-\n\tLogon Hours:\t\tAll\n\nAdditional Information:\n\tPrivileges\t\t-",
    "user": {
      "domain": "XXXXXXXXXXXXX",
      "name": "XXXXXXXX",
      "id": "S-1-5-18",
      "target": {
        "domain": "XXXXXXXXX",
        "name": "lenovo_tmp_ygauCNFY",
        "id": "S-1-5-21-2366788312-814031040-1301217162-1086"
      }
    },
    "tags": [
      "forwarded"
    ]
  },
  "fields": {
    "winlog.event_data.UserAccountControl": [
      "2080",
      "2082",
      "2084"
    ],
    "winlog.event_data.SidHistory": [
      "-"
    ],
    "event.category": [
      "iam"
    ],
    "winlog.event_data.NewUACList": [
      "LOCKOUT",
      "SCRIPT"
    ],
    "winlog.provider_guid": [
      "{54849625-5478-4994-a5ba-3e3b0328c30d}"
    ],
    "winlog.provider_name": [
      "Microsoft-Windows-Security-Auditing"
    ],
    "winlog.computer_name": [
      "XXXXXXXX.XXXXXXXXXXXXX.XXX"
    ],
    "user.target.id": [
      "S-1-5-21-2366788312-814031040-1301217162-1086"
    ],
    "winlog.process.pid": [
      672
    ],
    "winlog.event_data.UserParameters": [
      "%%1793"
    ],
    "winlog.keywords": [
      "Audit Success"
    ],
    "winlog.record_id": [
      "529299"
    ],
    "winlog.logon.id": [
      "0x3e7"
    ],
    "log.level": [
      "information"
    ],
    "agent.name": [
      "XXXXXXXX"
    ],
    "winlog.event_data.AllowedToDelegateTo": [
      "-"
    ],
    "winlog.event_data.HomePath": [
      "%%1793"
    ],
    "host.name": [
      "XXXXXXXX.XXXXXXXXXXXX.XXX"
    ],
    "user.target.name.text": [
      "lenovo_tmp_ygauCNFY"
    ],
    "event.kind": [
      "event"
    ],
    "winlog.activity_id": [
      "{41b78ff5-d8e1-0002-f890-b741e1d8da01}"
    ],
    "event.outcome": [
      "success"
    ],
    "winlog.event_data.OldUacValue": [
      "0x0"
    ],
    "winlog.event_data.UserWorkstations": [
      "%%1793"
    ],
    "winlog.event_data.TargetUserName": [
      "lenovo_tmp_ygauCNFY"
    ],
    "user.id": [
      "S-1-5-18"
    ],
    "agent.hostname": [
      "XXXXXXX"
    ],
    "related.user": [
      "XXXXXXXXXXX",
      "lenovo_tmp_ygauCNFY"
    ],
    "tags": [
      "forwarded"
    ],
    "user.target.name": [
      "lenovo_tmp_ygauCNFY"
    ],
    "event.provider": [
      "Microsoft-Windows-Security-Auditing"
    ],
    "event.code": [
      "4720"
    ],
    "agent.id": [
      "82295858-5992-45e2-bce1-966e78fb8d42"
    ],
    "ecs.version": [
      "1.12.0"
    ],
    "event.created": [
      "2024-07-30T18:51:10.110Z"
    ],
    "winlog.event_data.ScriptPath": [
      "%%1793"
    ],
    "agent.version": [
      "8.0.0"
    ],
    "winlog.event_data.SubjectUserSid": [
      "S-1-5-18"
    ],
    "winlog.process.thread.id": [
      1096
    ],
    "winlog.event_data.PrivilegeList": [
      "-"
    ],
    "user.name": [
      "XXXXXXXXX"
    ],
    "winlog.event_data.SamAccountName": [
      "lenovo_tmp_ygauCNFY"
    ],
    "winlog.event_data.PrimaryGroupId": [
      "513"
    ],
    "winlog.event_data.AccountExpires": [
      "%%1794"
    ],
    "winlog.event_data.DisplayName": [
      "%%1793"
    ],
    "agent.type": [
      "winlogbeat"
    ],
    "winlog.event_data.SubjectLogonId": [
      "0x3e7"
    ],
    "winlog.event_data.TargetSid": [
      "S-1-5-21-2366788312-814031040-1301217162-1086"
    ],
    "winlog.event_data.ProfilePath": [
      "%%1793"
    ],
    "winlog.api": [
      "wineventlog"
    ],
    "winlog.event_data.NewUacValue": [
      "0x15"
    ],
    "user.target.domain": [
      "XXXXXXXXX"
    ],
    "user.domain": [
      "XXXXXXXXXXXXXX"
    ],
    "winlog.event_data.UserPrincipalName": [
      "-"
    ],
    "winlog.task": [
      "User Account Management"
    ],
    "winlog.event_data.HomeDirectory": [
      "%%1793"
    ],
    "winlog.event_data.SubjectUserName": [
      "XXXXXXXXXXX"
    ],
    "message": [
      "A user account was created.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tXXXXXXXX\n\tAccount Domain:\t\tXXXXXXXXXXXXXX\n\tLogon ID:\t\t0x3E7\n\nNew Account:\n\tSecurity ID:\t\tS-1-5-21-2366788312-814031040-1301217162-1086\n\tAccount Name:\t\tlenovo_tmp_ygauCNFY\n\tAccount Domain:\t\tXXXXXXXXX\n\nAttributes:\n\tSAM Account Name:\tlenovo_tmp_ygauCNFY\n\tDisplay Name:\t\t<value not set>\n\tUser Principal Name:\t-\n\tHome Directory:\t\t<value not set>\n\tHome Drive:\t\t<value not set>\n\tScript Path:\t\t<value not set>\n\tProfile Path:\t\t<value not set>\n\tUser Workstations:\t<value not set>\n\tPassword Last Set:\t<never>\n\tAccount Expires:\t\t<never>\n\tPrimary Group ID:\t513\n\tAllowed To Delegate To:\t-\n\tOld UAC Value:\t\t0x0\n\tNew UAC Value:\t\t0x15\n\tUser Account Control:\t\n\t\tAccount Disabled\n\t\t'Password Not Required' - Enabled\n\t\t'Normal Account' - Enabled\n\tUser Parameters:\t<value not set>\n\tSID History:\t\t-\n\tLogon Hours:\t\tAll\n\nAdditional Information:\n\tPrivileges\t\t-"
    ],
    "winlog.event_id": [
      "4720"
    ],
    "winlog.event_data.LogonHours": [
      "%%1797"
    ],
    "event.action": [
      "added-user-account"
    ],
    "event.ingested": [
      "2024-07-30T18:51:10.223Z"
    ],
    "@timestamp": [
      "2024-07-30T18:50:59.547Z"
    ],
    "winlog.channel": [
      "Security"
    ],
    "event.type": [
      "user",
      "creation"
    ],
    "winlog.event_data.TargetDomainName": [
      "XXXXXXXXXX"
    ],
    "winlog.opcode": [
      "Info"
    ],
    "agent.ephemeral_id": [
      "fd18652b-adeb-49b1-a2e2-97239f97df8a"
    ],
    "winlog.event_data.PasswordLastSet": [
      "%%1794"
    ],
    "winlog.event_data.SubjectDomainName": [
      "XXXXXXXXXXXX"
    ],
    "user.name.text": [
      "XXXXXXXXX"
    ]
  }
}

```

@FlyNavy Please format your code by putting 3 backticks ````` before and after your code

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 30, 2024, 7:23pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/11 "2024-07-30T19:23:28Z")

</div>

what is a sample account mapping document?

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 7:32pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/12 "2024-07-30T19:32:33Z")

</div>

It would be mapping:  
winlog.event\_data.TargetSid to winlog.event\_data.TargetSID  
And  
winlog.event\_data.MemberName to winlog.event\_data.MemberName

The winlog.event\_data.MemberName is the missing element from the alert generating event. It's "-" in the above sample.

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 30, 2024, 7:42pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/13 "2024-07-30T19:42:03Z")

</div>

Sorry not tracking...

Here is my complete code... the problem is this...

```auto
"processors": [
 {
   "enrich": {
     "policy_name": "lenovo_account_enrichment_policy",
     "field": "user.target.id", <<<<. This field does not exist in your source document... 
     "target_field": "related.user_info",
     "max_matches": "1"
   }
 },

```

Your Doc has either

`user.id`  
or  
`user.target.group.id`

but not

` "field": "user.target.id",` which is in your pipeline does not exist in the incoming source document so it is not matching...

```auto
    "user": {
      "domain": "XXXXXXXXXX",
      "name": "XXXXXXXX",
      "id": "S-1-5-18", <<<<< this Field
      "target": {
        "name": "-",
        "group": {
          "domain": "Builtin",
          "name": "Administrators",
          "id": "S-1-5-32-544" <<<<< this Field
        }
      }
    },

```

My Complete code...

```auto
PUT /lenovo_account_mappings
{
"mappings": {
 "properties": {
   "sid": { "type": "keyword" },
   "account_name": { "type": "keyword" }
 }
}
}

POST lenovo_account_mappings/_doc
{
  "sid": "S-1-5-32-544",
  "account_name" : "brown"
}

PUT /_enrich/policy/lenovo_account_enrichment_policy
{
  "match": {
    "indices": "lenovo_account_mappings",
    "match_field": "sid",
    "enrich_fields": ["account_name"]
  }
}

PUT /_enrich/policy/lenovo_account_enrichment_policy/_execute?wait_for_completion=false

PUT _ingest/pipeline/enrich_alert_events
{
  "processors": [
    {
      "enrich": {
        "policy_name": "lenovo_account_enrichment_policy",
        "field": "user.target.group.id", <<< NOTE 
        "target_field": "related.user_info",
        "max_matches": "1"
      }
    }
  ]
}

POST _ingest/pipeline/enrich_alert_events/_simulate
{
  "docs": [
    {
      "_source": {
        "agent": {
          "name": "XXXXX",
          "id": "82295858-5992-45e2-bce1-966e78fb8d42",
          "type": "winlogbeat",
          "ephemeral_id": "fd18652b-adeb-49b1-a2e2-97239f97df8a",
          "version": "8.0.0"
        },
        "@timestamp": "2024-07-30T18:50:59.567Z",
        "winlog": {
          "computer_name": "XXXX.XXXXX.XXXXXX",
          "process": {
            "pid": 672,
            "thread": {
              "id": 1096
            }
          },
          "keywords": [
            "Audit Success"
          ],
          "logon": {
            "id": "0x3e7"
          },
          "channel": "Security",
          "event_data": {
            "SubjectUserName": "XXXXXXX",
            "MemberSid": "S-1-5-21-2366788312-814031040-1301217162-1086",
            "TargetSid": "S-1-5-32-544",
            "SubjectDomainName": "XXXXXXXX",
            "SubjectLogonId": "0x3e7",
            "MemberName": "-",
            "TargetUserName": "Administrators",
            "TargetDomainName": "Builtin",
            "SubjectUserSid": "S-1-5-18",
            "PrivilegeList": "-"
          },
          "opcode": "Info",
          "record_id": "529303",
          "task": "Security Group Management",
          "event_id": "4732",
          "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
          "activity_id": "{41b78ff5-d8e1-0002-f890-b741e1d8da01}",
          "api": "wineventlog",
          "provider_name": "Microsoft-Windows-Security-Auditing"
        },
        "ecs": {
          "version": "1.12.0"
        },
        "related": {
          "user": [
            "-",
            "XXXXXXXX"
          ]
        },
        "log": {
          "level": "information"
        },
        "host": {
          "name": "XXXXXX.XXXXXX.XXX"
        },
        "event": {
          "ingested": "2024-07-30T18:51:33.724955676Z",
          "code": "4732",
          "provider": "Microsoft-Windows-Security-Auditing",
          "created": "2024-07-30T18:51:33.147Z",
          "kind": "event",
          "action": "added-member-to-group",
          "category": [
            "iam"
          ],
          "type": [
            "group",
            "change"
          ],
          "outcome": "success"
        },
        "message": "A member was added to a security-enabled local group.\n\nSubject:\n\tSecurity ID:\t\tS-1-5-18\n\tAccount Name:\t\tXXXXXXXXX\n\tAccount Domain:\t\tXXXXXXXXXX\n\tLogon ID:\t\t0x3E7\n\nMember:\n\tSecurity ID:\t\tS-1-5-21-2366788312-814031040-1301217162-1086\n\tAccount Name:\t\t-\n\nGroup:\n\tSecurity ID:\t\tS-1-5-32-544\n\tGroup Name:\t\tAdministrators\n\tGroup Domain:\t\tBuiltin\n\nAdditional Information:\n\tPrivileges:\t\t-",
        "user": {
          "domain": "XXXXXXXXXX",
          "name": "XXXXXXXX",
          "id": "S-1-5-18",
          "target": {
            "name": "-",
            "group": {
              "domain": "Builtin",
              "name": "Administrators",
              "id": "S-1-5-32-544"
            }
          }
        },
        "tags": [
          "forwarded"
        ],
        "group": {
          "domain": "Builtin",
          "name": "Administrators",
          "id": "S-1-5-32-544"
        }
      }
    }
  ]
}

```

Result

```auto
...
         },
          "related": {
            "user_info": {
              "account_name": "brown",
              "sid": "S-1-5-32-544"
            },
            "user": [
              "-",
              "XXXXXXXX"
            ]
          },
  

```

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 7:48pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/14 "2024-07-30T19:48:21Z")

</div>

OK. Let me update those fields and run another simulation here to see if I get the right answer.

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 8:06pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/15 "2024-07-30T20:06:08Z")

</div>

So I updated the post to enter data into the custom index

```auto
POST lenovo_account_mappings/_doc
{
  "sid": "S-1-5-21-2366788312-814031040-1301217162-1086",
  "account_name": "Lenovo_tmp_ej9034hqwa"
}

```

I then updated the enrich alert events with the below to align with the data elements above.

```auto
PUT _ingest/pipeline/enrich_alert_events
{
  "processors": [
    {
      "enrich": {
        "policy_name": "lenovo_account_enrichment_policy",
        "field": "winlog.event_data.TargetSid",
        "target_field": "winlog.event_data.MemberName",
        "max_matches": "1"
      }
    }
  ]
}

```

I still get memberName equal to "-" when running the simulation

```auto
POST _ingest/pipeline/enrich_alert_events/_simulate
{
  "docs": [
    {
      "_source": {
        "agent": {
          "name": "XXXXX",
          "id": "82295858-5992-45e2-bce1-966e78fb8d42",
          "type": "winlogbeat",
          "ephemeral_id": "fd18652b-adeb-49b1-a2e2-97239f97df8a",
          "version": "8.0.0"
        },
        "@timestamp": "2024-07-30T18:50:59.567Z",
        "winlog": {
          "computer_name": "XXXX.XXXXX.XXXXXX",
          "process": {
            "pid": 672,
            "thread": {
              "id": 1096
            }
          },
          "keywords": [
            "Audit Success"
          ],
          "logon": {
            "id": "0x3e7"
          },
          "channel": "Security",
          "event_data": {
            "SubjectUserName": "XXXXXXX",
            "MemberSid": "S-1-5-21-2366788312-814031040-1301217162-1086",
            "TargetSid": "S-1-5-32-544",
            "SubjectDomainName": "XXXXXXXX",
            "SubjectLogonId": "0x3e7",
            "MemberName": "-",
            "TargetUserName": "Administrators",
            "TargetDomainName": "Builtin",
            "SubjectUserSid": "S-1-5-18",
            "PrivilegeList": "-"
          },
          "opcode": "Info",
          "record_id": "529303",
          "task": "Security Group Management",
          "event_id": "4732",
          "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
          "activity_id": "{41b78ff5-d8e1-0002-f890-b741e1d8da01}",
          "api": "wineventlog",
          "provider_name": "Microsoft-Windows-Security-Auditing"
        },
        "ecs": {
          "version": "1.12.0"
        },
        "related": {
          "user": [
            "-",
            "XXXXXXXX"
          ]
        },
        "log": {
          "level": "information"
        },
        "host": {
          "name": "XXXXXX.XXXXXX.XXX"
        },
        "event": {
          "ingested": "2024-07-30T18:51:33.724955676Z",
          "code": "4732",
          "provider": "Microsoft-Windows-Security-Auditing",
          "created": "2024-07-30T18:51:33.147Z",
          "kind": "event",
          "action": "added-member-to-group",
          "category": [
            "iam"
          ],
          "type": [
            "group",
            "change"
          ],
          "outcome": "success"
        },
        "message": """A member was added to a security-enabled local group.

Subject:
	Security ID: S-1-5-18
	Account Name: XXXXXXXXX
	Account Domain: XXXXXXXXXX
	Logon ID: 0x3E7

Member:
	Security ID: S-1-5-21-2366788312-814031040-1301217162-1086
	Account Name: -

Group:
	Security ID: S-1-5-32-544
	Group Name: Administrators
	Group Domain: Builtin

Additional Information:
	Privileges: -""",
        "user": {
          "domain": "XXXXXXXXXX",
          "name": "XXXXXXXX",
          "id": "S-1-5-18",
          "target": {
            "name": "-",
            "group": {
              "domain": "Builtin",
              "name": "Administrators",
              "id": "S-1-5-32-544"
            }
          }
        },
        "tags": [
          "forwarded"
        ],
        "group": {
          "domain": "Builtin",
          "name": "Administrators",
          "id": "S-1-5-32-544"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 8:08pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/16 "2024-07-30T20:08:43Z")

</div>

But I also don't think I have applied the enrichment policy to the ingestion pipelines yet.

---

<div class="post-metadata">

**Author:** ![FlyNavy](https://avatars.discourse-cdn.com/v4/letter/f/94ad74/32.png) [@FlyNavy](https://discuss.elastic.co/u/FlyNavy)\
**Post date:** [July 30, 2024, 8:15pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/17 "2024-07-30T20:15:15Z")

</div>

I think my reply got deleted.

I updated my custom index with the event SID and MemberName

```auto
POST lenovo_account_mappings/_doc
{
  "sid": "S-1-5-21-2366788312-814031040-1301217162-1086",
  "account_name": "Lenovo_tmp_ej9034hqwa"
}

```

I then updated the enrich policy with the below to align with the proper field names.

```auto
PUT _ingest/pipeline/enrich_alert_events
{
  "processors": [
    {
      "enrich": {
        "policy_name": "lenovo_account_enrichment_policy",
        "field": "winlog.event_data.TargetSid",
        "target_field": "winlog.event_data.MemberName",
        "max_matches": "1"
      }
    }
  ]
}

```

When running the simulation, I still get a blank "-" MemberName field in the simulation

```auto
POST _ingest/pipeline/enrich_alert_events/_simulate
{
  "docs": [
    {
      "_source": {
        "agent": {
          "name": "XXXXX",
          "id": "82295858-5992-45e2-bce1-966e78fb8d42",
          "type": "winlogbeat",
          "ephemeral_id": "fd18652b-adeb-49b1-a2e2-97239f97df8a",
          "version": "8.0.0"
        },
        "@timestamp": "2024-07-30T18:50:59.567Z",
        "winlog": {
          "computer_name": "XXXX.XXXXX.XXXXXX",
          "process": {
            "pid": 672,
            "thread": {
              "id": 1096
            }
          },
          "keywords": [
            "Audit Success"
          ],
          "logon": {
            "id": "0x3e7"
          },
          "channel": "Security",
          "event_data": {
            "SubjectUserName": "XXXXXXX",
            "MemberSid": "S-1-5-21-2366788312-814031040-1301217162-1086",
            "TargetSid": "S-1-5-32-544",
            "SubjectDomainName": "XXXXXXXX",
            "SubjectLogonId": "0x3e7",
            "MemberName": "-",
            "TargetUserName": "Administrators",
            "TargetDomainName": "Builtin",
            "SubjectUserSid": "S-1-5-18",
            "PrivilegeList": "-"
          },
          "opcode": "Info",
          "record_id": "529303",
          "task": "Security Group Management",
          "event_id": "4732",
          "provider_guid": "{54849625-5478-4994-a5ba-3e3b0328c30d}",
          "activity_id": "{41b78ff5-d8e1-0002-f890-b741e1d8da01}",
          "api": "wineventlog",
          "provider_name": "Microsoft-Windows-Security-Auditing"
        },
        "ecs": {
          "version": "1.12.0"
        },
        "related": {
          "user": [
            "-",
            "XXXXXXXX"
          ]
        },
        "log": {
          "level": "information"
        },
        "host": {
          "name": "XXXXXX.XXXXXX.XXX"
        },
        "event": {
          "ingested": "2024-07-30T18:51:33.724955676Z",
          "code": "4732",
          "provider": "Microsoft-Windows-Security-Auditing",
          "created": "2024-07-30T18:51:33.147Z",
          "kind": "event",
          "action": "added-member-to-group",
          "category": [
            "iam"
          ],
          "type": [
            "group",
            "change"
          ],
          "outcome": "success"
        },
        "message": """A member was added to a security-enabled local group.

Subject:
	Security ID: S-1-5-18
	Account Name: XXXXXXXXX
	Account Domain: XXXXXXXXXX
	Logon ID: 0x3E7

Member:
	Security ID: S-1-5-21-2366788312-814031040-1301217162-1086
	Account Name: -

Group:
	Security ID: S-1-5-32-544
	Group Name: Administrators
	Group Domain: Builtin

Additional Information:
	Privileges: -""",
        "user": {
          "domain": "XXXXXXXXXX",
          "name": "XXXXXXXX",
          "id": "S-1-5-18",
          "target": {
            "name": "-",
            "group": {
              "domain": "Builtin",
              "name": "Administrators",
              "id": "S-1-5-32-544"
            }
          }
        },
        "tags": [
          "forwarded"
        ],
        "group": {
          "domain": "Builtin",
          "name": "Administrators",
          "id": "S-1-5-32-544"
        }
      }
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 30, 2024, 8:38pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/18 "2024-07-30T20:38:49Z")

</div>

> [@FlyNavy](#):
>
> But I also don't think I have applied the enrichment policy to the ingestion pipelines yet.

Right because winlogbeat has its own ingest pipeline... you will need to add your enrich processor to the end of the OOTB pipeline...

So a couple questions..

Why 8.0.0 which is super old?

Why Winlogbeat maybe Elastic Agent would be better ...

Orr if you get on a newer version of the Stack and Winlogbeat there might be an easier way....

But if you just want to get it going add it to the bottom of the winlogbeat ingest pipeline (of course you will need to re-add if you upgrade)

You are posting too fast as a new user that is why you are getting throttled

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 30, 2024, 8:41pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/19 "2024-07-30T20:41:48Z")

</div>

`_source`

```
    "field": "winlog.event_data.TargetSid",

```

> [@FlyNavy](#):
>
> ` "TargetSid": "S-1-5-32-544",`

Lookup data

> [@FlyNavy](#):
>
> ` "sid": "S-1-5-21-2366788312-814031040-1301217162-1086",`

Not sure how you expect those to match... that values need to be the same for the match and enrich to happen

Looks like you used `MemberSid` .... you just need to get things lined up looks like your pipeline should be

` "field": "winlog.event_data.MemberSid",`

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 31, 2024, 1:56pm UTC](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084/20 "2024-07-31T13:56:27Z")

</div>

@FlyNavy

Did you get it working?

[Next page](https://discuss.elastic.co/t/enriching-data-with-custom-index/364084.md?page=2)