# Searching Duplicate terms in Index based on ID, splits the Id with limiter "-"

**URL:** <https://discuss.elastic.co/t/searching-duplicate-terms-in-index-based-on-id-splits-the-id-with-limiter/204900>\
**Category:** Elasticsearch\
**Created:** [October 23, 2019, 3:20pm UTC](https://discuss.elastic.co/t/searching-duplicate-terms-in-index-based-on-id-splits-the-id-with-limiter/204900 "2019-10-23T15:20:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vinothkumar\_Ganeshan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinothkumar_ganeshan/32/46534_2.png) [@Vinothkumar\_Ganeshan](https://discuss.elastic.co/u/Vinothkumar_Ganeshan)\
**Post date:** [October 23, 2019, 3:20pm UTC](https://discuss.elastic.co/t/searching-duplicate-terms-in-index-based-on-id-splits-the-id-with-limiter/204900/1 "2019-10-23T15:20:49Z")

</div>

Hello, I'm trying to find the duplicate based on a term "TDID" in a index.  
Basically the TDID : "00000-00000-0000000-00000"

> GET ttd-2019-04\*/\_search  
> {  
> "size" : 0,  
> "\_source": ["TDID"],  
> "query": {
> 
> ```
> "term": {
> "eventName": "impressions"
> }
> 
> ```
> 
> },  
> "aggs":{  
> "duplicate\_aggs":{  
> "terms":{  
> "field":"TDID",  
> "min\_doc\_count":2  
> }  
> }}  
> }

I got result as,  
"agg regations" : {  
"duplicate\_aggs" : {  
"doc\_count\_error\_upper\_bound" : 5448,  
"sum\_other\_doc\_count" : 77814191,  
"buckets" : [  
{  
"key" : "0000",  
"doc\_count" : 261766  
},  
{  
"key" : "00000000",  
"doc\_count" : 261542  
},  
{  
"key" : "000000000000",  
"doc\_count" : 261542  
},  
{  
"key" : "4cc0",  
"doc\_count" : 3540  
}}

As you can see, that the TDID is splitted and then aggreation was done. Could you please help me with that. Thanks for the support

---

<div class="post-metadata">

**Author:** ![William\_Brafford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/william_brafford/32/51559_2.png) [@William\_Brafford](https://discuss.elastic.co/u/William_Brafford)\
**Post date:** [October 23, 2019, 9:07pm UTC](https://discuss.elastic.co/t/searching-duplicate-terms-in-index-based-on-id-splits-the-id-with-limiter/204900/2 "2019-10-23T21:07:22Z")

</div>

Vinothkumar,

Are you using any special mappings for these indexes? Which version of Elasticsearch are you running?

My suspicion is that your `TDID` field is mapped as a [text datatype](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/text.html). A text field's contents will be split into tokens for searching. The mapping you want for an identifier like this is probably a [keyword datatype](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/keyword.html), which stores exact values like email addresses or phone numbers efficiently for searching.

You _might_ have a keyword field as a default mapping. If so, you could use the following aggregation:

```auto
{
  [...],
  "aggs": {
    "duplicate_aggs": {
      "terms": {
        "field": "key.keyword",
        "min_doc_count": 2
      }
    }
  }
}

```

If that doesn't work, the correct answer will depend on which Elasticsearch version you are running and what your index's mappings are.

-William

---

<div class="post-metadata">

**Author:** ![Vinothkumar\_Ganeshan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vinothkumar_ganeshan/32/46534_2.png) [@Vinothkumar\_Ganeshan](https://discuss.elastic.co/u/Vinothkumar_Ganeshan)\
**Post date:** [October 24, 2019, 8:28am UTC](https://discuss.elastic.co/t/searching-duplicate-terms-in-index-based-on-id-splits-the-id-with-limiter/204900/3 "2019-10-24T08:28:28Z")

</div>

@William_Brafford Thanks for your response. Above works, as I could see the duplicates. A Clarification, if possible. I need to create a new document based on grouping(Merging) by the TDID with specific fields to combine and create a document to be inserted as index.

In Simple: All the documents should be based on individual TDID, where its eventname like impression, clicks are combined as list in the same document.

Index: tdid-based  
Doc:  
field: TDID : 000-000-03251-0000  
field: events: ['Clicks', 'impression']

So, I could see that the TDID with different events

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2019, 8:35am UTC](https://discuss.elastic.co/t/searching-duplicate-terms-in-index-based-on-id-splits-the-id-with-limiter/204900/4 "2019-11-21T08:35:31Z")

</div>

Closed on November 21, 2019, 8:35am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.