# Only keep the most recent line when the id is the same

**URL:** <https://discuss.elastic.co/t/only-keep-the-most-recent-line-when-the-id-is-the-same/130640>\
**Category:** Elasticsearch\
**Created:** [May 4, 2018, 1:57pm UTC](https://discuss.elastic.co/t/only-keep-the-most-recent-line-when-the-id-is-the-same/130640 "2018-05-04T13:57:58Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pierre2](https://avatars.discourse-cdn.com/v4/letter/p/da6949/32.png) [@Pierre2](https://discuss.elastic.co/u/Pierre2)\
**Post date:** [May 4, 2018, 1:57pm UTC](https://discuss.elastic.co/t/only-keep-the-most-recent-line-when-the-id-is-the-same/130640/1 "2018-05-04T13:57:59Z")

</div>

Hello,

My logs lines can have the same ID. If it's the case I want to keep only the most recent line of them.  
For exemple if I have 3 lines with these fields :

> ID : 2568 | a : toto | b : tata | @timestamp : 4 may 2018 00:00:00

> ID : 2568 | a : momo| b : titi | @timestamp : 4 may 2018 05:00:00

> ID : 2568 | a : thyt| b : drgfr| @timestamp : 4 may 2018 16:00:00

In this example the 3 lines have the same ID so I only want the most recent of them so :

> ID : 2568 | a: thyt| b: drgfr| @timestamp : 4 may 2018 16:00:00

I don't know how to proceed... Should I use the agreggate filter ? If it's the case, how ?

Thanks

---

<div class="post-metadata">

**Author:** ![ddorian43](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddorian43/32/36093_2.png) [@ddorian43](https://discuss.elastic.co/u/ddorian43)\
**Post date:** [May 4, 2018, 1:59pm UTC](https://discuss.elastic.co/t/only-keep-the-most-recent-line-when-the-id-is-the-same/130640/2 "2018-05-04T13:59:36Z")

</div>

See `field collapsing` [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-collapse.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-collapse.html)

---

<div class="post-metadata">

**Author:** ![Pierre2](https://avatars.discourse-cdn.com/v4/letter/p/da6949/32.png) [@Pierre2](https://discuss.elastic.co/u/Pierre2)\
**Post date:** [May 4, 2018, 2:12pm UTC](https://discuss.elastic.co/t/only-keep-the-most-recent-line-when-the-id-is-the-same/130640/3 "2018-05-04T14:12:17Z")

</div>

I want to do this in logstash and only keep the most recent lines and drop the others. I want to do this verification every x seconds. I do not that it is possible with field collapsing, am I right ?

---

<div class="post-metadata">

**Author:** ![ddorian43](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ddorian43/32/36093_2.png) [@ddorian43](https://discuss.elastic.co/u/ddorian43)\
**Post date:** [May 4, 2018, 2:25pm UTC](https://discuss.elastic.co/t/only-keep-the-most-recent-line-when-the-id-is-the-same/130640/4 "2018-05-04T14:25:01Z")

</div>

Yes. Collapsing is only for querying. Looks like you want to delete, which I don't know how.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 1, 2018, 2:35pm UTC](https://discuss.elastic.co/t/only-keep-the-most-recent-line-when-the-id-is-the-same/130640/5 "2018-06-01T14:35:20Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.