# Logstash: filter unique key documents

**URL:** <https://discuss.elastic.co/t/logstash-filter-unique-key-documents/313880>\
**Category:** Logstash\
**Created:** [September 7, 2022, 11:58am UTC](https://discuss.elastic.co/t/logstash-filter-unique-key-documents/313880 "2022-09-07T11:58:49Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Moshe\_Sharon1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moshe_sharon1/32/110058_2.png) [@Moshe\_Sharon1](https://discuss.elastic.co/u/Moshe_Sharon1)\
**Post date:** [September 7, 2022, 11:58am UTC](https://discuss.elastic.co/t/logstash-filter-unique-key-documents/313880/1 "2022-09-07T11:58:49Z")

</div>

Hello,

We have an index which has multiple documents with the same phone number.

Each document will always contain the phone number and may contain additional information (see example below)

The documents are written into the index constantly.

For a certain time window, for example 1 minute, we would like to have only one record for each unique phone number.

This data is sent via Logstash into an S3 bucket.

We do not care which record will be selected (first, last, random) but we need only one.

Input index example:

| phone no | time | data | address | … | … |
| --- | --- | --- | --- | --- | --- |
| 50-5325471 | 1:10:50 | A | | | |
| 50-5325471 | 1:10:51 | B | | | |
| 50-5325471 | 1:10:52 | C | | | |
| 55-6789345 | 1:10:50 | A | | | |
| 55-6789345 | 1:10:53 | B | | | |
| 57-3434345 | 1:10:55 | C | | | |
| 50-5325471 | 1:15:50 | D | | | |
| 50-5325471 | 1:15:55 | E | | | |
| 55-6789345 | 1:15:50 | F | | | |

Output data example:

| phone no | time | data | address | … | … |
| --- | --- | --- | --- | --- | --- |
| 50-5325471 | 1:10:50 | A | | | |
| 55-6789345 | 1:10:50 | A | | | |
| 50-5325471 | 1:15:55 | E | | | |
| 55-6789345 | 1:15:50 | F | | | |

The idea is to have a query that will filter one record per phone number for each Logstash iteration.

Can someone please help on the way to do that.

Thanks

**From:** Moshe Sharon \<[moshe.sharon@cellwize.com](mailto:moshe.sharon@cellwize.com)\>  
**Sent:** Wednesday, 7 September 2022 12:02  
**To:** Tomer Bruchiel \<[tomer.bruchiel@cellwize.com](mailto:tomer.bruchiel@cellwize.com)\>  
**Subject:** Please check and correct

**Logstash Filter problem**

We have an index which has multiple documents with the same phone number.

ex: table showing index example

| phone no | time | name | address | … | … |
| --- | --- | --- | --- | --- | --- |
| 50-5325471 | | | | | |
| 50-5325471 | | | | | |
| 50-5325471 | | | | | |
| 55-6789345 | | | | | |
| 55-6789345 | | | | | |
| 57-3434345 | | | | | |
| 50-5325471 | | | | | |
| 50-5325471 | | | | | |
| 55-6789345 | | | | | |

I want Logstash to write as output the latest(or first) occurrence of same phone no from every batch (same phone no returning one after the other)

thanks

Moshe  
[sharon.moshe@gmail.com](mailto:sharon.moshe@gmail.com)

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 7, 2022, 12:22pm UTC](https://discuss.elastic.co/t/logstash-filter-unique-key-documents/313880/2 "2022-09-07T12:22:16Z")

</div>

You can use [Fingerprint](https://www.elastic.co/guide/en/logstash/current/plugins-filters-fingerprint.html) plugin to calculate unique value which will used for document\_id  
Something like this should work if time is format hh:mm, not with seconds. In one minute you will have insert A, update B, update C. At the end, you will have only the last record like data - C, as update.

```auto
filter{
...
    fingerprint {
       source => ["phone no","time"]
       concatenate_sources => true
       target => "[@metadata][docid]"
       method => "UUID"
       id => "fingerprint" 
    }
...
}
output {
  elasticsearch {
  hosts => ["http://localhost:9200"]
  index => "phones"
  document_id => "%{[@metadata][docid]}"
  }
}

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [September 7, 2022, 3:20pm UTC](https://discuss.elastic.co/t/logstash-filter-unique-key-documents/313880/3 "2022-09-07T15:20:14Z")

</div>

Assuming you parsed the [time] field into [@timestamp] you can add a metadata field to use in the fingerprint that only has hours and minutes

```
mutate { add_field => { "[@metadata][time]" => "%{{HH:mm}}" } }

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [September 7, 2022, 8:32pm UTC](https://discuss.elastic.co/t/logstash-filter-unique-key-documents/313880/4 "2022-09-07T20:32:49Z")

</div>

You can also try with [aggregations](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html)

---

<div class="post-metadata">

**Author:** ![Moshe\_Sharon1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/moshe_sharon1/32/110058_2.png) [@Moshe\_Sharon1](https://discuss.elastic.co/u/Moshe_Sharon1)\
**Post date:** [September 8, 2022, 1:08pm UTC](https://discuss.elastic.co/t/logstash-filter-unique-key-documents/313880/5 "2022-09-08T13:08:37Z")

</div>

Hi Rios  
Thank you

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 6, 2022, 1:09pm UTC](https://discuss.elastic.co/t/logstash-filter-unique-key-documents/313880/6 "2022-10-06T13:09:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.