# Duplicates Query not returning all results

**URL:** <https://discuss.elastic.co/t/duplicates-query-not-returning-all-results/58600>\
**Category:** Elasticsearch\
**Created:** [August 22, 2016, 5:17pm UTC](https://discuss.elastic.co/t/duplicates-query-not-returning-all-results/58600 "2016-08-22T17:17:52Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![johns](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@johns](https://discuss.elastic.co/u/johns)\
**Post date:** [August 22, 2016, 5:17pm UTC](https://discuss.elastic.co/t/duplicates-query-not-returning-all-results/58600/1 "2016-08-22T17:17:52Z")

</div>

I am searching for and counting duplicated phrases within a single, or group of, human readable documents. I break each document into phrases/sentences and populate an Elasticsearch index with these phrases, one per ES document.

I have 707 documents in my index. I KNOW that I should have, at least, 25 duplicate documents. My search is returning 23 duplicate docs. I don't understand why I am missing some matches.

Notes:  
I am creating my index with 1 shard and 0 replicas.  
I am matching all string content in a field so I am md5 hashing the field value.

Here is my query:

```
{
    "size": 0,
    "aggs": {
        "duplicateCount": {
            "terms": {
                "field": "content",
                "min_doc_count": 2
            },
            "aggs": {
                "duplicateDocuments": {
                    "top_hits": {

                    }
                }
            }
        }
    }
}
```

My process:

1. Create index
2. Build bulk insert data objects
3. Bulk insert documents into index
4. Reindex documents
5. Run duplicates query (above)
6. Parse results - SUM buckets.doc\_counts
7. delete index

Why is ES not returning all duplicates????

Thanks

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [August 23, 2016, 10:45am UTC](https://discuss.elastic.co/t/duplicates-query-not-returning-all-results/58600/2 "2016-08-23T10:45:58Z")

</div>

The `terms` aggregation by default returns only 10 values so in your case max 10 md5s. For these 10 hashes you may have only 23 docs that share these keys.  
If you use the `size` setting on the terms agg you can increase the number of md5s under consideration.

---

<div class="post-metadata">

**Author:** ![johns](https://avatars.discourse-cdn.com/v4/letter/j/87869e/32.png) [@johns](https://discuss.elastic.co/u/johns)\
**Post date:** [August 23, 2016, 2:37pm UTC](https://discuss.elastic.co/t/duplicates-query-not-returning-all-results/58600/3 "2016-08-23T14:37:43Z")

</div>

You're the Man Mark! Thanks so much - that fixed it.  
My new query:

```
{
    "size": 0,
    "aggs": {
        "duplicateCount": {
            "terms": {
                "size": 708,
                "field": "content",
                "min_doc_count": 2
            },
            "aggs": {
                "duplicateDocuments": {
                    "top_hits": {

                    }
                }
            }
        }
    }
}
```

Since I always know how many docs I have I always keep my query reasonably efficient. When I hit 10000 I'll have to figure out how to use scroll 🙂

Thanks again,  
John

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:25pm UTC](https://discuss.elastic.co/t/duplicates-query-not-returning-all-results/58600/4 "2017-07-05T22:25:51Z")

</div>