Only keep the most recent line when the id is the same


#1

Hello,

My logs lines can have the same ID. If it's the case I want to keep only the most recent line of them.
For exemple if I have 3 lines with these fields :

ID : 2568 | a : toto | b : tata | @timestamp : 4 may 2018 00:00:00

ID : 2568 | a : momo| b : titi | @timestamp : 4 may 2018 05:00:00

ID : 2568 | a : thyt| b : drgfr| @timestamp : 4 may 2018 16:00:00

In this example the 3 lines have the same ID so I only want the most recent of them so :

ID : 2568 | a: thyt| b: drgfr| @timestamp : 4 may 2018 16:00:00

I don't know how to proceed... Should I use the agreggate filter ? If it's the case, how ?

Thanks


(ddorian43) #2

See field collapsing https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-collapse.html


#3

I want to do this in logstash and only keep the most recent lines and drop the others. I want to do this verification every x seconds. I do not that it is possible with field collapsing, am I right ?


(ddorian43) #4

Yes. Collapsing is only for querying. Looks like you want to delete, which I don't know how.


(system) #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.