Delete duplicate docs in ES 1.7

[muhamadli302]

Hello,

I'm having a problem finding the right query for searching and deleting duplicated documents in my index.
I'm using ES 1.7.

Thank you for your help

[mainec (Isabel Drost-Fromm)]

How do you define a duplicate?

A brief search across the history of the discuss forum I'm currently typing this answer in revealed the following threads which might be helpful for you:

How to filter out duplicate documents across multiple types?

We have certain documents stored across multiple types with translated values, for example, US and ES types has same document but with different values in title fields. Example: US: { "title":"Manning: Spring in Action, Third Edition" } ES: { "title":"Manning : Primavera en Acción , Tercera Edición" } So, when I search for "Manning" across all types, I only want one document. I can certainly remove the duplicates in my code, but then I can not use pagination. Any one know how to r…

Filebeat, Logstash, Elasticsearch robustness and duplicated documents

For logging in AWS EC2, I'm testing the robustness of the chain Filebeat, Logstash, Elasticsearch. I have one AMI with an appplication + Filebeat, one with Logstash and one with Elastisearch + Kibana. With the application running I try to reboot one of these 3 machines and see what happens when it's back available. The good news is that I never loose any line of log. The less good one is that most of the times I end up having duplicated logs in Elasticsearch. Typically I generate X lines (let's…

Using elasticsearch to find duplicates in dataset

Hello, I am currently evaluating elasticsearch for a very specific task which is removing duplicates from a contacts list, from my initial tests it looks like it would works but there are still some shadows I hope you can help me with. What I was planning to do is: load the data from some csv files normalize the fields (phone numbers, addresses) load the data into elasticsearch run a bunch of queries on the data to find/remove/merge the duplicates export the data back into csv The first …

Is there any way to remove duplicated search result in ES?

Thank you for your rapid reply . it is true that i can custom my own search action, but i can not override the default search action .so, it is not what i want. at indexing time , there are serval listeners to install plugins, but at searching time there is hardly any listener to extend the search operation except the search action . why not provide a opportunity to install my own plugin to extend the search phase , because it seems to be simple from the source code . i should give up th…

Hope this helps to get you started,
Isabel

[muhamadli302]

Hi,
Thank you for your replay,
my definition for a duplicate is the same document with the same unique field under the same index.
This situation is caused by problems in inserting the data. So now I need to find an efficient way to find all the duplicates and delete them (leaving only one copy). It would be best if you could help us create a query that will find those duplicates and delete them.

Thank you!