# Find and delete duplicate documents in Elasticsearch

**URL:** <https://discuss.elastic.co/t/find-and-delete-duplicate-documents-in-elasticsearch/231533>\
**Category:** Elasticsearch\
**Created:** [May 7, 2020, 12:11pm UTC](https://discuss.elastic.co/t/find-and-delete-duplicate-documents-in-elasticsearch/231533 "2020-05-07T12:11:18Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Youssef\_SBAI](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/youssef_sbai/32/64242_2.png) [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\
**Post date:** [May 7, 2020, 12:11pm UTC](https://discuss.elastic.co/t/find-and-delete-duplicate-documents-in-elasticsearch/231533/1 "2020-05-07T12:11:18Z")

</div>

I use python for find and delete duplicate documents in Elasticsearch

I use this script but i have an error

```
#!/usr/local/bin/python3
import hashlib
from elasticsearch import Elasticsearch
es = Elasticsearch(["localhost:9200"])
dict_of_duplicate_docs = {}
# La ligne suivante définit les champs qui seront
# utilisés pour déterminer si un document est un doublon ou non
keys_to_include_in_hash = ["chaine", "job", "date_plan", "statut", "date_debut", "date_fin", "serveur", "numero_passage", "application", "sous_application"]
# Traitement des documents renvoyés par la recherche actuelle ou le défilement
def populate_dict_of_duplicate_docs(hits):
    for item in hits:
        combined_key = ""
        for mykey in keys_to_include_in_hash:
            combined_key += str(item['_source'][mykey])
        _id = item["_id"]
        hashval = hashlib.md5(combined_key.encode('utf-8')).digest()
        # Si la valeur du hachage (hashval) est nouvelle, nous créerons une nouvelle clé
        # dans dict_of_duplicate_docs, qui se verra attribuer
        # une valeur d’un tableau vide.
        # Nous intégrerons ensuite immédiatement la valeur _id dans le tableau.
        # Si la valeur hashval existe déjà, alors
        # nous indiquerons tout simplement la nouvelle valeur _id dans le tableau existant.
        dict_of_duplicate_docs.setdefault(hashval, []).append(_id)
# Créez une boucle avec tous les documents de l’index, puis renseignez la
# structure de données dict_of_duplicate_docs.
def scroll_over_all_docs():
    data = es.search(index="hello", scroll='1m', body={"query": {"match_all": {}}})
    # Obtenez l’ID de défilement
    sid = data['_scroll_id']
    scroll_size = len(data['hits']['hits'])
    # Avant de procéder au défilement, traitez le lot de résultats en cours
    populate_dict_of_duplicate_docs(data['hits']['hits'])
    while scroll_size > 0:
        data = es.scroll(scroll_id=sid, scroll='2m')
        # Traitez le lot de résultats en cours
        populate_dict_of_duplicate_docs(data['hits']['hits'])
        # Mettez l’ID de défilement à jour
        sid = data['_scroll_id']
        # Obtenez le nombre de résultats renvoyés lors du dernier défilement
        scroll_size = len(data['hits']['hits'])
def loop_over_hashes_and_remove_duplicates():
    # Parcourez les valeurs de hachage pour déterminer s’il existe
    # des hachages en double
    for hashval, array_of_ids in dict_of_duplicate_docs.items():
      if len(array_of_ids) > 1:
        print(" **********Duplicate docs hash=%s**********" % hashval)
        # Obtenez les documents mappés à la valeur hashval actuelle
        matching_docs = es.mget(index="hello", doc_type="doc", body={"ids": array_of_ids})
        for doc in matching_docs['docs']:
            # Dans cet exemple, nous avons simplement imprimé les documents en double.
            # Il est tout à fait possible (et simple !) de modifier ce code pour supprimer les doublons
            # ici au lieu de les imprimer.
            print("doc=%s\n" % doc)
def main():
    scroll_over_all_docs()
    loop_over_hashes_and_remove_duplicates()
main()
```

---

<div class="post-metadata">

**Author:** ![Youssef\_SBAI](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/youssef_sbai/32/64242_2.png) [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\
**Post date:** [May 7, 2020, 12:11pm UTC](https://discuss.elastic.co/t/find-and-delete-duplicate-documents-in-elasticsearch/231533/2 "2020-05-07T12:11:41Z")

</div>

` <urllib3.connection.HTTPConnection object at 0x7f526dc264e0>: Failed to establish a new connection: [Errno 111] Connection refused`

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 7, 2020, 11:36pm UTC](https://discuss.elastic.co/t/find-and-delete-duplicate-documents-in-elasticsearch/231533/3 "2020-05-07T23:36:30Z")

</div>

Can you `curl -H "Content-Type: application/json" -XGET localhost:9200/`?

---

<div class="post-metadata">

**Author:** ![Youssef\_SBAI](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/youssef_sbai/32/64242_2.png) [@Youssef\_SBAI](https://discuss.elastic.co/u/Youssef_SBAI)\
**Post date:** [May 9, 2020, 10:04am UTC](https://discuss.elastic.co/t/find-and-delete-duplicate-documents-in-elasticsearch/231533/4 "2020-05-09T10:04:09Z")

</div>

` curl -H "Content-Type: application/json" -XGET localhost:9200/?`

```
{
  "name" : "C06513361",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "dyFy1g9AQvWuI4E5Sd8odQ",
  "version" : {
    "number" : "7.5.2",
    "build_flavor" : "default",
    "build_type" : "zip",
    "build_hash" : "8bec50e1e0ad29dad5653712cf3bb580cd1afcdf",
    "build_date" : "2020-01-15T12:11:52.313576Z",
    "build_snapshot" : false,
    "lucene_version" : "8.3.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [May 10, 2020, 11:03pm UTC](https://discuss.elastic.co/t/find-and-delete-duplicate-documents-in-elasticsearch/231533/5 "2020-05-10T23:03:11Z")

</div>

Seems odd that it would give you a `Connection refused` error then?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 7, 2020, 11:03pm UTC](https://discuss.elastic.co/t/find-and-delete-duplicate-documents-in-elasticsearch/231533/6 "2020-06-07T23:03:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.