Remove documents with existing value in given field


I am trying to delete or pipe (Delete), an event that already has the same indexed value for a given field.

The scenario is as follows:

I have a fleet server and an agent to fetch logs from AD Server, Windows fires a lot of events just for a user login, that's noise to me and I was wondering if I would create a pipeline or a renderer (drop_event) in the policy of system processors.

the point is that each triggered event has a unique LogonGUID, I want to discard the event that is already stored in Elasticsearch with the same LogonGUID value.

Just to get one document per login.

I was wondering if anyone can help by doing something like

if stored document.LogonGUID === next document.LogonGUID


Set up some pipeline to process that before indexing.

Thank you!


An simple solution is using the LogonGUID as the _id of the document of Elasticsearch.
Elasticsearch automatically overwrite the same docoment with the same LogonGUID.

Yes, it sounds like a solution, but not a complete one, in AD the logonGUID is something like this:


There is another one that is like the previous one but filled with 0.


I was missing making an exception with this, but if I used it as _id, maybe I lose some information. But I will definitely try.

this field LogonGUID, it's not a really "unique" id for logon, a user logon generates maybe 2 of this but its betten than 5 or 6.


Then, how to decide to update or not to update?

By default windows doesn't have that option, just made replacement for the _id and works better than nothing.


1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.