# Can i prevent duplicate entry in logstash template?

**URL:** <https://discuss.elastic.co/t/can-i-prevent-duplicate-entry-in-logstash-template/160098>\
**Category:** Logstash\
**Created:** [December 10, 2018, 5:31am UTC](https://discuss.elastic.co/t/can-i-prevent-duplicate-entry-in-logstash-template/160098 "2018-12-10T05:31:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![inandi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inandi/32/40801_2.png) [@inandi](https://discuss.elastic.co/u/inandi)\
**Post date:** [December 10, 2018, 5:31am UTC](https://discuss.elastic.co/t/can-i-prevent-duplicate-entry-in-logstash-template/160098/1 "2018-12-10T05:31:50Z")

</div>

I am using the template to insert data to ES. But some duplicate value is inserting, I want to prevent them baseed on **reviewId** in my example

**sample data**

```
[
  {
    "reviewId": "1",
    "displayName": "JOHN"
  },
  {
    "reviewId": "2",
    "displayName": "MAYA"
  },
  {
    "reviewId": "1",
    "displayName": "JOHN"
  }
] 

```

my **template** is

```
{
  "index_patterns": ["data_my_*"],
  "settings": {
    "index": {
      "number_of_shards": "2",
      "number_of_replicas": "1"
    }
  },
  "mappings": {
    "doc":{
      "dynamic": "false",
      "properties": {
        "reviewId": {
          "type": "text"
        }, 
        "displayName": {
          "type": "text"
        }
      }
    }
  },
  "aliases": {}
}

```

**Logstash conf** file

```
input {
  file {
    path => "/home/templetes/dt5.txt"
    start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => "json"        
  }
}

filter { 
  json {
    source => "message"
  } 
}

output {
  if [reviewId] != "" {
    stdout { codec => rubydebug }
    elasticsearch {
      hosts => ["http://127.0.0.1:9200"] 
      index => "data_my_1"
      template =>"/home/templetes/datamobi.json"
      template_name=>"datamobi"
      template_overwrite => true
    }
  }
}

```

thanks, !!!

---

<div class="post-metadata">

**Author:** ![Mojster](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mojster/32/21209_2.png) [@Mojster](https://discuss.elastic.co/u/Mojster)\
**Post date:** [December 10, 2018, 12:53pm UTC](https://discuss.elastic.co/t/can-i-prevent-duplicate-entry-in-logstash-template/160098/2 "2018-12-10T12:53:39Z")

</div>

You should use `document_id` in the output.  
Here's an article on duplicates:

> **[Little Logstash Lessons: Handling Duplicates](https://www.elastic.co/blog/logstash-lessons-handling-duplicates)**
>
> Approaches for de-duplicating data in Elasticsearch using Logstash. We also go into examples of how you can use IDs in Elasticsearch Output.

---

<div class="post-metadata">

**Author:** ![inandi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/inandi/32/40801_2.png) [@inandi](https://discuss.elastic.co/u/inandi)\
**Post date:** [December 11, 2018, 5:08am UTC](https://discuss.elastic.co/t/can-i-prevent-duplicate-entry-in-logstash-template/160098/3 "2018-12-11T05:08:36Z")

</div>

thanks @Mojster ... thumbs up

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 8, 2019, 5:13am UTC](https://discuss.elastic.co/t/can-i-prevent-duplicate-entry-in-logstash-template/160098/4 "2019-01-08T05:13:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.