# Remove documents with existing value in given field

**URL:** <https://discuss.elastic.co/t/remove-documents-with-existing-value-in-given-field/306572>\
**Category:** Elasticsearch\
**Created:** [June 7, 2022, 1:23pm UTC](https://discuss.elastic.co/t/remove-documents-with-existing-value-in-given-field/306572 "2022-06-07T13:23:46Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![fran\_hernandez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fran_hernandez/32/106728_2.png) [@fran\_hernandez](https://discuss.elastic.co/u/fran_hernandez)\
**Post date:** [June 7, 2022, 1:23pm UTC](https://discuss.elastic.co/t/remove-documents-with-existing-value-in-given-field/306572/1 "2022-06-07T13:23:46Z")

</div>

Hello,

I am trying to delete or pipe (Delete), an event that already has the same indexed value for a given field.

The scenario is as follows:

I have a fleet server and an agent to fetch logs from AD Server, Windows fires a lot of events just for a user login, that's noise to me and I was wondering if I would create a pipeline or a renderer (drop\_event) in the policy of system processors.

the point is that each triggered event has a unique LogonGUID, I want to discard the event that is already stored in Elasticsearch with the same LogonGUID value.

Just to get one document per login.

I was wondering if anyone can help by doing something like

```auto
drop_event
if stored document.LogonGUID === next document.LogonGUID

```

EITHER

Set up some pipeline to process that before indexing.

Thank you!

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [June 8, 2022, 8:42am UTC](https://discuss.elastic.co/t/remove-documents-with-existing-value-in-given-field/306572/2 "2022-06-08T08:42:05Z")

</div>

Hi,

An simple solution is using the LogonGUID as the \_id of the document of Elasticsearch.  
Elasticsearch automatically overwrite the same docoment with the same LogonGUID.

---

<div class="post-metadata">

**Author:** ![fran\_hernandez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fran_hernandez/32/106728_2.png) [@fran\_hernandez](https://discuss.elastic.co/u/fran_hernandez)\
**Post date:** [June 8, 2022, 9:33am UTC](https://discuss.elastic.co/t/remove-documents-with-existing-value-in-given-field/306572/3 "2022-06-08T09:33:44Z")

</div>

Yes, it sounds like a solution, but not a complete one, in AD the logonGUID is something like this:

{B9E4085B-6641-0F85-D5F4-0C04EF81725B}

There is another one that is like the previous one but filled with 0.

{00000000-00000...}

I was missing making an exception with this, but if I used it as \_id, maybe I lose some information. But I will definitely try.

this field LogonGUID, it's not a really "unique" id for logon, a user logon generates maybe 2 of this but its betten than 5 or 6.

Thanks!

---

<div class="post-metadata">

**Author:** ![Tomo\_M](https://avatars.discourse-cdn.com/v4/letter/t/848f3c/32.png) [@Tomo\_M](https://discuss.elastic.co/u/Tomo_M)\
**Post date:** [June 8, 2022, 10:44am UTC](https://discuss.elastic.co/t/remove-documents-with-existing-value-in-given-field/306572/4 "2022-06-08T10:44:44Z")

</div>

Then, how to decide to update or not to update?

---

<div class="post-metadata">

**Author:** ![fran\_hernandez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fran_hernandez/32/106728_2.png) [@fran\_hernandez](https://discuss.elastic.co/u/fran_hernandez)\
**Post date:** [June 8, 2022, 1:09pm UTC](https://discuss.elastic.co/t/remove-documents-with-existing-value-in-given-field/306572/5 "2022-06-08T13:09:33Z")

</div>

By default windows doesn't have that option, just made replacement for the \_id and works better than nothing.

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2022, 1:10pm UTC](https://discuss.elastic.co/t/remove-documents-with-existing-value-in-given-field/306572/6 "2022-07-06T13:10:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.