Remove documents with existing value in given field

[fran_hernandez]

Hello,

I am trying to delete or pipe (Delete), an event that already has the same indexed value for a given field.

The scenario is as follows:

I have a fleet server and an agent to fetch logs from AD Server, Windows fires a lot of events just for a user login, that's noise to me and I was wondering if I would create a pipeline or a renderer (drop_event) in the policy of system processors.

the point is that each triggered event has a unique LogonGUID, I want to discard the event that is already stored in Elasticsearch with the same LogonGUID value.

Just to get one document per login.

I was wondering if anyone can help by doing something like

drop_event
if stored document.LogonGUID === next document.LogonGUID

EITHER

Set up some pipeline to process that before indexing.

Thank you!

[Tomo_M (Tomohiro Mitani)]

Hi,

An simple solution is using the LogonGUID as the _id of the document of Elasticsearch.
Elasticsearch automatically overwrite the same docoment with the same LogonGUID.

[fran_hernandez]

Yes, it sounds like a solution, but not a complete one, in AD the logonGUID is something like this:

{B9E4085B-6641-0F85-D5F4-0C04EF81725B}

There is another one that is like the previous one but filled with 0.

{00000000-00000...}

I was missing making an exception with this, but if I used it as _id, maybe I lose some information. But I will definitely try.

this field LogonGUID, it's not a really "unique" id for logon, a user logon generates maybe 2 of this but its betten than 5 or 6.

Thanks!

[Tomo_M (Tomohiro Mitani)]

Then, how to decide to update or not to update?

[fran_hernandez]

By default windows doesn't have that option, just made replacement for the _id and works better than nothing.

Thanks.