# Find duplicate docs by multi fields

**URL:** <https://discuss.elastic.co/t/find-duplicate-docs-by-multi-fields/120352>\
**Category:** Elasticsearch\
**Created:** [February 18, 2018, 1:16pm UTC](https://discuss.elastic.co/t/find-duplicate-docs-by-multi-fields/120352 "2018-02-18T13:16:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Guylot](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guylot/32/47141_2.png) [@Guylot](https://discuss.elastic.co/u/Guylot)\
**Post date:** [February 18, 2018, 1:16pm UTC](https://discuss.elastic.co/t/find-duplicate-docs-by-multi-fields/120352/1 "2018-02-18T13:16:09Z")

</div>

Hi,

I need to find duplicate docs which is determined by multi fields,  
and I want to run this operation daily.  
Right now I have 2 solutions:

1. Script query where I concate the fields into one field and do term aggregation on it.

> ```
> {
> "size": 0,
> "aggs": {
> "duplicate_docs": {
> "terms": {
> "script": "params['_source']['id1'] + params['_source']['id2'] + params['_source']['id3'] + params['_source']['id4'] + params['_source']['id5'] + params['_source']['date']",
> "min_doc_count": 2,
> "size": 1000
> },
> "aggs": {
> "dup_docs": {
> "top_hits": {
> "size": 5
> }
> }
> }
> }
> }
> }
> 
> ```

I used the doc[field].value first but I got some wrong results (I gusse because of the analyzer) so I switched to \_source.  
I also have a date field that I concate. Can this field cause performance problems when joining it to normal string?  
This operation seems to be a little bit heavy on cpu and memory depends on the number of docs I have.

2, I read about copy\_field but it looks like A waste to create A concated field only for monitoring needs.

- I also tried nested aggregation but its to complicate and I dont think its a good solution.

Is there any other way to find duplicates? Or anyway to improve the script?

Thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 18, 2018, 1:16pm UTC](https://discuss.elastic.co/t/find-duplicate-docs-by-multi-fields/120352/2 "2018-03-18T13:16:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.