# Duplicate data in logstash

**URL:** <https://discuss.elastic.co/t/duplicate-data-in-logstash/59758>\
**Category:** Logstash\
**Created:** [September 4, 2016, 11:35am UTC](https://discuss.elastic.co/t/duplicate-data-in-logstash/59758 "2016-09-04T11:35:13Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mikail\_Land](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@Mikail\_Land](https://discuss.elastic.co/u/Mikail_Land)\
**Post date:** [September 4, 2016, 11:35am UTC](https://discuss.elastic.co/t/duplicate-data-in-logstash/59758/1 "2016-09-04T11:35:13Z")

</div>

i have duplicate data in logstash  
my config file is :

```
input {
  file {
    path => "/var/log/flask/access*"
    type => "flask_access"
    max_open_files => 409599
  }
  stdin{}
}
filter {
  mutate { replace => { "type" => "flask_access" } }
  grok {
    match => { "message" => "%{FLASKACCESS}" }
  }
  mutate {
    add_field => {
      "temp" => "%{uniqueid} %{method}"
    }
  }
   if "Entering" in [api_status] {
     aggregate {
       task_id => "%{temp}"
       code => "map['blockedprocess'] = 2"
       map_action => "create"
     }
   }
   if "Entering" in [api_status] or "Leaving" in [api_status]{
     aggregate {
       task_id => "%{temp}"
       code => "map['blockedprocess'] -= 1"
       map_action => "update"
     }
   }
   if "End Task" in [api_status] {
     aggregate {
       task_id => "%{temp}"
       code => "event['blockedprocess'] = map['blockedprocess']"
       map_action => "update"
       end_of_task => true
       timeout => 120
     }
   }
 }
output {
  elasticsearch {
    hosts => ["localhost:9200"]
# sniffing => true
# manage_template => false
# index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
# document_type => "%{[@metadata][type]}"
  }
  stdout { codec => rubydebug }
}

```

please help me  
thanks

---

<div class="post-metadata">

**Author:** ![Mikail\_Land](https://avatars.discourse-cdn.com/v4/letter/m/82dd89/32.png) [@Mikail\_Land](https://discuss.elastic.co/u/Mikail_Land)\
**Post date:** [September 4, 2016, 1:24pm UTC](https://discuss.elastic.co/t/duplicate-data-in-logstash/59758/2 "2016-09-04T13:24:39Z")

</div>

i solve it

i create a unique id by ('document\_id') in output section

document\_id point to my temp and temp is my unique id in my project

my output changed to:

```
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    document_id => "%{temp}"
# sniffing => true
# manage_template => false
# index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
# document_type => "%{[@metadata][type]}"
  }
  stdout { codec => rubydebug }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:40am UTC](https://discuss.elastic.co/t/duplicate-data-in-logstash/59758/3 "2017-07-06T04:40:01Z")

</div>