Duplicate documents in elaticsearch

[hiba (chaieb)]

Hi,
i have below configuration of logstash.
input {
http_poller {
urls => {
url => "XXXX"
}
request_timeout => 60
schedule => { every => "60s" }
codec => "json"
}
}
filter {
split { field => "[bookings]" }
split { field => "[bookings][rooms]" }
mutate {
rename => {
"[bookings][status]" => "status"
"[bookings][hotelId]" => "hotelId"
"[bookings][hotelName]" => "hotelName"
"[bookings][hotelCity]" => "hotelCity"
"[bookings][hotelCountry]" => "hotelCountry"
"[bookings][arrDate]" => "arrDate"
"[bookings][depDate]" => "depDate"
"[bookings][price]" => "price"
"[bookings][currency]" => "currency"
"[bookings][purchasePrice]" => "purchasePrice"
"[bookings][partnerName]" => "partnerName"
"[bookings][partnerId]" => "partnerId"
"[bookings][firstName]" => "firstName"
"[bookings][lastName]" => "lastName"
"[bookings][channel]" => "channel"
"[bookings][supplierName]" => "supplierName"
"[bookings][rooms][board]" => "board"
"[bookings][rooms][paxes][adults]" => "adults"
"[bookings][rooms][paxes][infant]" => "infant"
"[bookings][rooms][paxes][children]" => "children"
"[bookings][rooms][quantity]" => "quantity"
"[bookings][rooms][room]" => "room"
}
remove_field => ["bookingId", "confirmedDate", "bookingRef", "bookings", "createdDate", "hotelAddress", "hotelPhonearrDate", "customerId", "title", "email", "city", "mobile", "supplierId","paxe", "payments", "options", "isXML"]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
index => "bookings"
}
stdout { codec => rubydebug }
}

so when i tested the json response with "json editor", i got 25 knot

but after the execution I had 93hits in "discover kibana"

any help please !

[warkolm (Mark Walkom)]

@hiba

schedule => { every => "60s" }

It's because of this line, you are polling the API every minute and so it will be retrieving the records again and uploading them to Elasticsearch. Elasticsearch sees them as new documents though, as the _id value will be different.

If you would like to deduplicate them then you will need to define your own _id in the elasticsearch output section, using something unique. I don't know your use case, so it's hard to recommend anything else on that at the moment.