# Duplicate documents in elaticsearch

**URL:** <https://discuss.elastic.co/t/duplicate-documents-in-elaticsearch/231057>\
**Category:** Logstash\
**Created:** [May 5, 2020, 12:10am UTC](https://discuss.elastic.co/t/duplicate-documents-in-elaticsearch/231057 "2020-05-05T00:10:48Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![hiba](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hiba/32/110919_2.png) [@hiba](https://discuss.elastic.co/u/hiba)\
**Post date:** [May 5, 2020, 12:10am UTC](https://discuss.elastic.co/t/duplicate-documents-in-elaticsearch/231057/1 "2020-05-05T00:10:49Z")

</div>

Hi,  
i have below configuration of logstash.  
input {  
http\_poller {  
urls =\> {  
url =\> "XXXX"  
}  
request\_timeout =\> 60  
schedule =\> { every =\> "60s" }  
codec =\> "json"  
}  
}  
filter {  
split { field =\> "[bookings]" }  
split { field =\> "[bookings][rooms]" }  
mutate {  
rename =\> {  
"[bookings][status]" =\> "status"  
"[bookings][hotelId]" =\> "hotelId"  
"[bookings][hotelName]" =\> "hotelName"  
"[bookings][hotelCity]" =\> "hotelCity"  
"[bookings][hotelCountry]" =\> "hotelCountry"  
"[bookings][arrDate]" =\> "arrDate"  
"[bookings][depDate]" =\> "depDate"  
"[bookings][price]" =\> "price"  
"[bookings][currency]" =\> "currency"  
"[bookings][purchasePrice]" =\> "purchasePrice"  
"[bookings][partnerName]" =\> "partnerName"  
"[bookings][partnerId]" =\> "partnerId"  
"[bookings][firstName]" =\> "firstName"  
"[bookings][lastName]" =\> "lastName"  
"[bookings][channel]" =\> "channel"  
"[bookings][supplierName]" =\> "supplierName"  
"[bookings][rooms][board]" =\> "board"  
"[bookings][rooms][paxes][adults]" =\> "adults"  
"[bookings][rooms][paxes][infant]" =\> "infant"  
"[bookings][rooms][paxes][children]" =\> "children"  
"[bookings][rooms][quantity]" =\> "quantity"  
"[bookings][rooms][room]" =\> "room"   
}  
remove\_field =\> ["bookingId", "confirmedDate", "bookingRef", "bookings", "createdDate", "hotelAddress", "hotelPhonearrDate", "customerId", "title", "email", "city", "mobile", "supplierId","paxe", "payments", "options", "isXML"]  
}  
}  
output {  
elasticsearch {  
hosts =\> "localhost:9200"  
index =\> "bookings"  
}  
stdout { codec =\> rubydebug }  
}

so when i tested the json response with "json editor", i got 25 knot  
 ![Capturetest](https://us1.discourse-cdn.com/elastic/original/3X/e/0/e0f8d653d88954fd88f94afb6cd468fc88936bfc.png)

but after the execution I had 93hits in "discover kibana"

 ![Capture.3PNG](https://us1.discourse-cdn.com/elastic/original/3X/2/1/21335e3c05444da188c6b3c143aa31f81dc03afc.png) ![Capture2](https://us1.discourse-cdn.com/elastic/original/3X/3/0/309c18ae640e86fd57ffdc8bb855f2f4d9e3ce2a.png)

any help please !

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 5, 2020, 5:56am UTC](https://discuss.elastic.co/t/duplicate-documents-in-elaticsearch/231057/2 "2020-05-05T05:56:56Z")

</div>

> [@hiba](#):
>
> schedule =\> { every =\> "60s" }

It's because of this line, you are polling the API every minute and so it will be retrieving the records again and uploading them to Elasticsearch. Elasticsearch sees them as new documents though, as the `_id` value will be different.

If you would like to deduplicate them then you will need to define your own `_id` in the `elasticsearch` output section, using something unique. I don't know your use case, so it's hard to recommend anything else on that at the moment.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 2, 2020, 5:57am UTC](https://discuss.elastic.co/t/duplicate-documents-in-elaticsearch/231057/3 "2020-06-02T05:57:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.