# Uax\_url\_email tokenizer unexpected result

**URL:** <https://discuss.elastic.co/t/uax-url-email-tokenizer-unexpected-result/172694>\
**Category:** Elasticsearch\
**Created:** [March 17, 2019, 11:20pm UTC](https://discuss.elastic.co/t/uax-url-email-tokenizer-unexpected-result/172694 "2019-03-17T23:20:00Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![doaks](https://avatars.discourse-cdn.com/v4/letter/d/67e7ee/32.png) [@doaks](https://discuss.elastic.co/u/doaks)\
**Post date:** [March 17, 2019, 11:20pm UTC](https://discuss.elastic.co/t/uax-url-email-tokenizer-unexpected-result/172694/1 "2019-03-17T23:20:00Z")

</div>

I define a sub-field like so:

```
        "tokenizer": {
            "my_email_tokenizer": {
                "type": "uax_url_email",
                "max_token_length": 100,
            }
        },
        ....
            "my_email_analyzer": {
                "type": "custom",
                "tokenizer": "my_email_tokenizer",
                "filter": ["lowercase", "stop","length_filter"]
            },
            ...
            "fields": {
                "emails":{
                   "type":"text",
                   "analyzer":"my_email_analyzer", 
                },

```

However when I try and analyze the email "foobar@baz.mail" against this field, the result is:

{'tokens': [{'end\_offset': 13,  
'position': 0,  
'start\_offset': 0,  
'token': 'foobar@baz.ma',  
'type': ''},  
{'end\_offset': 15,  
'position': 1,  
'start\_offset': 13,  
'token': 'il',  
'type': ''}]}

Why is it splitting up the mail token? I thought it might be the max length, but I set it to 100 to be sure.

I am using ES 6.3.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 20, 2019, 8:46am UTC](https://discuss.elastic.co/t/uax-url-email-tokenizer-unexpected-result/172694/2 "2019-03-20T08:46:18Z")

</div>

The tokenizer uses a list of TLDs and not just anything like `a@b.WHATEVER`, which sometimes needs some time to update. I dont know on top of my head if `.mail` is a valid TLD, if it is, this may require an update in lucene.

A quick google search shows that .mail is not approved by ICANN. I do not know the input source of the data in lucene though.

hope this helps!

---

<div class="post-metadata">

**Author:** ![doaks](https://avatars.discourse-cdn.com/v4/letter/d/67e7ee/32.png) [@doaks](https://discuss.elastic.co/u/doaks)\
**Post date:** [March 20, 2019, 9:15pm UTC](https://discuss.elastic.co/t/uax-url-email-tokenizer-unexpected-result/172694/3 "2019-03-20T21:15:30Z")

</div>

Does ICANN disallow local domains in email addresses?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 17, 2019, 9:15pm UTC](https://discuss.elastic.co/t/uax-url-email-tokenizer-unexpected-result/172694/4 "2019-04-17T21:15:42Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.