# duplicateNames with multiple fields?

**URL:** <https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209>\
**Category:** Elasticsearch\
**Created:** [February 16, 2018, 3:33pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209 "2018-02-16T15:33:38Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![liamwazherealso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liamwazherealso/32/27868_2.png) [@liamwazherealso](https://discuss.elastic.co/u/liamwazherealso)\
**Post date:** [February 16, 2018, 3:33pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209/1 "2018-02-16T15:33:38Z")

</div>

Hello, I'm able to preform a query and find duplicate results with.

```
query = {
    "aggs": {
        "duplicateNames": {
            "terms": {
                "field": comparison_field,
                "size": 0,
                "min_doc_count": 2
            }
        }
    }
}

```

However is there a way I can do this for multiple fields? Ensuring that the document has all of the fields duplicated?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 16, 2018, 3:45pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209/2 "2018-02-16T15:45:25Z")

</div>

I'd probably compute a "hash" field based on the sum of the values of other fields at index time, then I'd try to aggregate on it.

---

<div class="post-metadata">

**Author:** ![liamwazherealso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liamwazherealso/32/27868_2.png) [@liamwazherealso](https://discuss.elastic.co/u/liamwazherealso)\
**Post date:** [February 16, 2018, 4:07pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209/3 "2018-02-16T16:07:58Z")

</div>

That's a fine solution. But given what that would entail for my system I'd prefer a work around.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 16, 2018, 4:52pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209/4 "2018-02-16T16:52:06Z")

</div>

Then you can use that probably: [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-script](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#search-aggregations-bucket-terms-aggregation-script)

And emit a unique key based on the combination of the values you have in different fields.

I can predict that it will be dramatically slow but if speed is not an issue that might fit your needs.

---

<div class="post-metadata">

**Author:** ![liamwazherealso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liamwazherealso/32/27868_2.png) [@liamwazherealso](https://discuss.elastic.co/u/liamwazherealso)\
**Post date:** [February 16, 2018, 4:55pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209/5 "2018-02-16T16:55:28Z")

</div>

Speeds not an issue, will try this out, thanks!

---

<div class="post-metadata">

**Author:** ![liamwazherealso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liamwazherealso/32/27868_2.png) [@liamwazherealso](https://discuss.elastic.co/u/liamwazherealso)\
**Post date:** [February 19, 2018, 3:18pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209/7 "2018-02-19T15:18:25Z")

</div>

I'm coming up to an issue using this. Albeit it may not be related to the query. I'm thrown a `elasticsearch.exceptions.RequestError: <exception str() failed>`  
when I pass this query.

```
 {
       'aggs':{
          'duplicateNames':{
             'terms':{
                'script':"doc['@timestamp'].value"
             },
             'min_doc_count':2,
             'size':0
          }
       }
    }

```

Any idea on what this could be? I'm using [Elastic Search 2.4](https://www.elastic.co/guide/en/elasticsearch/reference/2.4/search-aggregations-bucket-terms-aggregation.html), so the syntax is a bit different.

edit: I believe the issue is that scripting is not enabled by default.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [February 19, 2018, 3:44pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209/8 "2018-02-19T15:44:51Z")

</div>

No.

Could you provide a full recreation script as described in [About the Elasticsearch category](https://discuss.elastic.co/t/about-the-elasticsearch-category/21). It will help to better understand what you are exactly doing. Please, try to keep the example as simple as possible.

---

<div class="post-metadata">

**Author:** ![liamwazherealso](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liamwazherealso/32/27868_2.png) [@liamwazherealso](https://discuss.elastic.co/u/liamwazherealso)\
**Post date:** [February 21, 2018, 7:07pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209/9 "2018-02-21T19:07:38Z")

</div>

I will do so for further questions. After some investigations I figured out that the reason it wasn't working was because scripting wasn't enabled.

I can't change that so I'm opting to try and use a solution mentioned [here](https://stackoverflow.com/a/21302068/1325202). If I can't get that to work I will be using your has idea.

Thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2018, 7:08pm UTC](https://discuss.elastic.co/t/duplicatenames-with-multiple-fields/120209/10 "2018-03-21T19:08:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.