Only keep the most recent line when the id is the same

Elastic Stack Elasticsearch
1

Hello,

My logs lines can have the same ID. If it's the case I want to keep only the most recent line of them.
For exemple if I have 3 lines with these fields :

ID : 2568 | a : toto | b : tata | @timestamp : 4 may 2018 00:00:00

ID : 2568 | a : momo| b : titi | @timestamp : 4 may 2018 05:00:00

ID : 2568 | a : thyt| b : drgfr| @timestamp : 4 may 2018 16:00:00

In this example the 3 lines have the same ID so I only want the most recent of them so :

ID : 2568 | a: thyt| b: drgfr| @timestamp : 4 may 2018 16:00:00

I don't know how to proceed... Should I use the agreggate filter ? If it's the case, how ?

Thanks

2

See field collapsing https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-collapse.html

3

I want to do this in logstash and only keep the most recent lines and drop the others. I want to do this verification every x seconds. I do not that it is possible with field collapsing, am I right ?

4

Yes. Collapsing is only for querying. Looks like you want to delete, which I don't know how.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.