Duplicate documents in elaticsearch

Hi,
i have below configuration of logstash.
input {
http_poller {
urls => {
url => "XXXX"
}
request_timeout => 60
schedule => { every => "60s" }
codec => "json"
}
}
filter {
split { field => "[bookings]" }
split { field => "[bookings][rooms]" }
mutate {
rename => {
"[bookings][status]" => "status"
"[bookings][hotelId]" => "hotelId"
"[bookings][hotelName]" => "hotelName"
"[bookings][hotelCity]" => "hotelCity"
"[bookings][hotelCountry]" => "hotelCountry"
"[bookings][arrDate]" => "arrDate"
"[bookings][depDate]" => "depDate"
"[bookings][price]" => "price"
"[bookings][currency]" => "currency"
"[bookings][purchasePrice]" => "purchasePrice"
"[bookings][partnerName]" => "partnerName"
"[bookings][partnerId]" => "partnerId"
"[bookings][firstName]" => "firstName"
"[bookings][lastName]" => "lastName"
"[bookings][channel]" => "channel"
"[bookings][supplierName]" => "supplierName"
"[bookings][rooms][board]" => "board"
"[bookings][rooms][paxes][adults]" => "adults"
"[bookings][rooms][paxes][infant]" => "infant"
"[bookings][rooms][paxes][children]" => "children"
"[bookings][rooms][quantity]" => "quantity"
"[bookings][rooms][room]" => "room"
}
remove_field => ["bookingId", "confirmedDate", "bookingRef", "bookings", "createdDate", "hotelAddress", "hotelPhonearrDate", "customerId", "title", "email", "city", "mobile", "supplierId","paxe", "payments", "options", "isXML"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "bookings"
}
stdout { codec => rubydebug }
}

so when i tested the json response with "json editor", i got 25 knot
Capturetest

but after the execution I had 93hits in "discover kibana"

Capture2

any help please !

It's because of this line, you are polling the API every minute and so it will be retrieving the records again and uploading them to Elasticsearch. Elasticsearch sees them as new documents though, as the _id value will be different.

If you would like to deduplicate them then you will need to define your own _id in the elasticsearch output section, using something unique. I don't know your use case, so it's hard to recommend anything else on that at the moment.