# Duplicate events ingested by m365\_defender module

**URL:** <https://discuss.elastic.co/t/duplicate-events-ingested-by-m365-defender-module/290901>\
**Category:** Elastic Security\
**Created:** [December 3, 2021, 1:25pm UTC](https://discuss.elastic.co/t/duplicate-events-ingested-by-m365-defender-module/290901 "2021-12-03T13:25:20Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Foxboron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/foxboron/32/98115_2.png) [@Foxboron](https://discuss.elastic.co/u/Foxboron)\
**Post date:** [December 3, 2021, 1:25pm UTC](https://discuss.elastic.co/t/duplicate-events-ingested-by-m365-defender-module/290901/1 "2021-12-03T13:25:20Z")

</div>

It seems like `m365_defender` creates duplicate events. This seems to be because of non-stable ordering of the document fields. In our data it seems like fields inside the `alert` object changes their order and some of the fields in `agent`. I think this is the primary issue which causes duplication.

Our idea is to then add some static fingerprinting to the ingest pipeline, and see if that solves the issue. But this isn't super trivial since it seems like Sentinel updates the read object. Thus we have the challenge of which fields to include which would deduplicate the events, but not collide with updated events.

We have two suggestions!

### Suggestion 1:

```auto
processors:
  - fingerprint:
      fields: ["microsoft.m365_defender.incidentUri"]
      target_field: "@metadata._id"

```

This would probably cause updated events to collide, so we we probably don't think this is a good idea. Along with the entity types changing.

### Suggestion 2

```auto
processors:
  - fingerprint:
      fields:
        - "microsoft.m365_defender.incidentUri"
        - "microsoft.m365_defender.alerts.providerAlertId"
        - "microsoft.m365_defender.alerts.entities.entityType"
      target_field: "@metadata._id"

```

This seems more sane, but the drawback is that we see some events being updated with new values in only one field. This fingerprinting would not allow us to collect the updated events.

It seems like fixing the ordering issue or the root cause of there being ordering issues in the data would be better. But it's hard to figure out where that happens.

---

<div class="post-metadata">

**Author:** ![spong](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spong/32/54343_2.png) [@spong](https://discuss.elastic.co/u/spong)\
**Post date:** [December 9, 2021, 11:19pm UTC](https://discuss.elastic.co/t/duplicate-events-ingested-by-m365-defender-module/290901/2 "2021-12-09T23:19:13Z")

</div>

Thanks for the feedback @Foxboron! And welcome to the community! 🙂 👋

I've forwarded along your feedback to the team, and also added it to [this issue](https://github.com/elastic/integrations/issues/936#issuecomment-990391756) for migrating the `M365 Defender Module` to an `Elastic Agent Integration`. Perhaps this can be addressed as part of that effort.

Cheers!  
Garrett

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 6, 2022, 11:19pm UTC](https://discuss.elastic.co/t/duplicate-events-ingested-by-m365-defender-module/290901/3 "2022-01-06T23:19:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.