# Duplicate Tokens in elasticsearch uax\_url\_email tokenizer

**URL:** <https://discuss.elastic.co/t/duplicate-tokens-in-elasticsearch-uax-url-email-tokenizer/301398>\
**Category:** Elasticsearch\
**Created:** [April 2, 2022, 7:02pm UTC](https://discuss.elastic.co/t/duplicate-tokens-in-elasticsearch-uax-url-email-tokenizer/301398 "2022-04-02T19:02:48Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheGreatBaldOne](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thegreatbaldone/32/93645_2.png) [@TheGreatBaldOne](https://discuss.elastic.co/u/TheGreatBaldOne)\
**Post date:** [April 2, 2022, 7:02pm UTC](https://discuss.elastic.co/t/duplicate-tokens-in-elasticsearch-uax-url-email-tokenizer/301398/1 "2022-04-02T19:02:48Z")

</div>

I have created a test index here

```auto
{
                    "settings" : {
                        "analysis" : {
                            "filter" : {
                            "emailcustom" : {
                                    "type" : "pattern_capture",
                                    "preserve_original" : true,
                                    "patterns" : [
                                        "([^@]+)",
                                        "(\\p{L}+)",
                                        "(\\d+)",
                                        "@(.+)"
                                    ]
                                }
                            },
                            "analyzer" : {
                                "my_email_analyzer" : {
                                    "tokenizer" : "uax_url_email",
                                    "filter" : ["lowercase", "asciifolding", "emailcustom"]
                                }
                            }
                        }
                    },
                    "mappings": {
                        "properties": {
			                "descwithemail":{
				                "type": "keyword", 
				                "fields": {
                                    "text": { 
                                        "type": "text",
						                "analyzer": "my_email_analyzer"
                                    }
                        }         
			          }          
                    }    
                }
            }

```

When I try and

```auto
GET /myindex/_analyze
             {
            "analyzer": "my_email_analyzer",
            "text": "My email address is john@domain.com"
            }

```

I get duplicate tokens namely '[domain.com](http://domain.com)' and 'john'

I know I should use the 'unique' filter but what is the impact of not using or using it.

I also do not understand where the duplicate tokens come from.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 30, 2022, 7:03pm UTC](https://discuss.elastic.co/t/duplicate-tokens-in-elasticsearch-uax-url-email-tokenizer/301398/2 "2022-04-30T19:03:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.