# Is it possible detect duplicate message in Kafka topic with logstash

**URL:** <https://discuss.elastic.co/t/is-it-possible-detect-duplicate-message-in-kafka-topic-with-logstash/193026>\
**Category:** Logstash\
**Created:** [July 31, 2019, 3:55am UTC](https://discuss.elastic.co/t/is-it-possible-detect-duplicate-message-in-kafka-topic-with-logstash/193026 "2019-07-31T03:55:50Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![arp220](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/arp220/32/84666_2.png) [@arp220](https://discuss.elastic.co/u/arp220)\
**Post date:** [July 31, 2019, 3:55am UTC](https://discuss.elastic.co/t/is-it-possible-detect-duplicate-message-in-kafka-topic-with-logstash/193026/1 "2019-07-31T03:55:50Z")

</div>

we use `logstash` and we want read one table from `oracle database` and send this message to `kafka`, we format message in `kafka` is:

```auto
Topic1: message1: {"name":"name-1", "id":"fbd89256-12gh-10og-etdgn1234njF", "site":"site-1", "time":"2019-07-30"}
        message2: {"name":"name-2", "id":"fbd89256-12gh-10og-etdgn1234njG", "site":"site-1", "time":"2019-07-30"}
        message3: {"name":"name-3", "id":"fbd89256-12gh-10og-etdgn1234njS", "site":"site-1", "time":"2019-07-30"}
        message4: {"name":"name-4", "id":"fbd89256-12gh-10og-etdgn1234njF", "site":"site-1", "time":"2019-07-30"}

```

Please note that `message1` and `message4` are the duplicates with the `same ID number`.

Now, we want sure all `messages` are `unique`, so how can we filter `topic1` and `unique` all message then `send` to `topic2`?

The end result we want:

```auto
Topic2: message1: {"name":"name-1", "id":"fbd89256-12gh-10og-etdgn1234njF", "site":"site-1", "time":"2019-07-30"}
        message2: {"name":"name-2", "id":"fbd89256-12gh-10og-etdgn1234njG", "site":"site-1", "time":"2019-07-30"}
        message3: {"name":"name-3", "id":"fbd89256-12gh-10og-etdgn1234njS", "site":"site-1", "time":"

```

Is this possible with logstash?  
thanks.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 31, 2019, 1:17pm UTC](https://discuss.elastic.co/t/is-it-possible-detect-duplicate-message-in-kafka-topic-with-logstash/193026/2 "2019-07-31T13:17:01Z")

</div>

It is possible

```
    mutate { add_field => { "[@metadata][constant]" => "task" } }
    aggregate {
        task_id => "%{[@metadata][constant]}"
        code => '
            map["ids"] ||= []
            id = event.get("id")
            if map["ids"].include? id
                event.cancel
            else
                map["ids"] << id
            end
        '
    }

```

However, that never purges old entries, so eventually it will run out of memory. Also, in current versions you cannot control whether you get name-1 or name-4.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 28, 2019, 1:17pm UTC](https://discuss.elastic.co/t/is-it-possible-detect-duplicate-message-in-kafka-topic-with-logstash/193026/3 "2019-08-28T13:17:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.