# Filter duplicate record of log in kibana dashboard **URL:** \ **Category:** Kibana\ **Created:** [June 14, 2020, 7:31am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016 "2020-06-14T07:31:48Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![anand27889](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@anand27889](https://discuss.elastic.co/u/anand27889)\ **Post date:** [June 14, 2020, 7:31am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/1 "2020-06-14T07:31:48Z")
Hi Friends, I want to remove the duplicate record of log file which is displaying in kibana dashboard, for example: {TransactionID: 23750 , Status: failed, Time: 14/06/2020 11:20:25} {TransactionID: 23750 , Status: Passed, Time: 14/06/2020 11:25:12} In the above example, both the transaction id are same but the latest status is passed. So I want to show only the last processed record. Kindly help me on this. Thanks. ---
**Author:** ![mattkime](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattkime/32/43522_2.png) [@mattkime](https://discuss.elastic.co/u/mattkime)\ **Post date:** [June 16, 2020, 6:07pm UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/2 "2020-06-16T18:07:24Z")
The solution depends upon how your data is structured and what question you wish to answer. You could just filter based on status, or perhaps you could aggregate on transaction ID. It might also make sense to have a single document instead of two. ---
**Author:** ![anand27889](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@anand27889](https://discuss.elastic.co/u/anand27889)\ **Post date:** [June 17, 2020, 6:50am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/3 "2020-06-17T06:50:00Z")
Hi @mattkime Thanks for your reply, I may have n number of records in my log file, there may be a change to get many duplicate entries in my file. I want to display the total number of records i received for a day with out duplicate record. The unique column is transaction id. if there is any duplicate, i need to pick only the recent record based on the timestamp. Kindly guide me. Thanks in Advance. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 17, 2020, 6:53am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/4 "2020-06-17T06:53:28Z")
Try using a top hits aggregation on the timestamp with a terms agg on the transaction ID. That should get it. ---
**Author:** ![anand27889](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@anand27889](https://discuss.elastic.co/u/anand27889)\ **Post date:** [June 17, 2020, 6:59am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/5 "2020-06-17T06:59:26Z")
Hi @warkolm I need to display the total transaction count in dashboard, if i use top hits aggregation on timestamp with term agg on transaction id, it is showing the last transaction received not the total count. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 17, 2020, 7:00am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/6 "2020-06-17T07:00:32Z")
Does that mean transaction IDs are not unique? ---
**Author:** ![anand27889](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@anand27889](https://discuss.elastic.co/u/anand27889)\ **Post date:** [June 17, 2020, 7:21am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/7 "2020-06-17T07:21:18Z")
@warkolm Transaction Id will be repeated in duplicate entries. For example: {TransactionID: 23751 , Status: Passed, Time: 14/06/2020 11:20:25} {TransactionID: 23752 , Status: Passed, Time: 14/06/2020 11:25:12} {TransactionID: 23753 , Status: failed, Time: 14/06/2020 11:30:25} {TransactionID: 23753 , Status: Passed, Time: 14/06/2020 11:35:12} ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 17, 2020, 7:22am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/8 "2020-06-17T07:22:51Z")
I'm a little confused here. You ask how to only show one record, the latest, then you talk about wanting to show the total count? Do you want both of those things or just one of them? ---
**Author:** ![anand27889](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@anand27889](https://discuss.elastic.co/u/anand27889)\ **Post date:** [June 17, 2020, 7:26am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/9 "2020-06-17T07:26:34Z")
@warkolm I just want to display the total transaction which i received for a day, I can process the same transaction again if it gets failed, so Transaction id will not change for that but one entry will be added in the log file. so kibana taking that record also in the count. I hope now you understand my requirement clearly. Thanks. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [June 17, 2020, 7:27am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/10 "2020-06-17T07:27:58Z")
Put another way - you want the latest transaction for each ID, then a total count all of those for the day? ---
**Author:** ![anand27889](https://avatars.discourse-cdn.com/v4/letter/a/258eb7/32.png) [@anand27889](https://discuss.elastic.co/u/anand27889)\ **Post date:** [June 17, 2020, 8:21am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/11 "2020-06-17T08:21:03Z")
@warkolm yes, exactly. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 15, 2020, 8:21am UTC](https://discuss.elastic.co/t/filter-duplicate-record-of-log-in-kibana-dashboard/237016/12 "2020-07-15T08:21:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.