# Email Tokenizing Not Working? **URL:** \ **Category:** Elasticsearch\ **Created:** [January 9, 2012, 4:33pm UTC](https://discuss.elastic.co/t/email-tokenizing-not-working/6329 "2012-01-09T16:33:21Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\ **Post date:** [January 9, 2012, 4:33pm UTC](https://discuss.elastic.co/t/email-tokenizing-not-working/6329/1 "2012-01-09T16:33:21Z")
Hi, I've defined an analyser and used it in a Mapping as defined below: > But when I enter something like @gmail.com, it is not matching the wildcard search, it only matches when I enter "test.user@gmail.com" or "test.user" or "test." etc. i.e. there appears to be no tokenization on the "@" or the ".". This is the query I'm using: > Regards, David. ---
**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\ **Post date:** [January 10, 2012, 9:46am UTC](https://discuss.elastic.co/t/email-tokenizing-not-working/6329/2 "2012-01-10T09:46:44Z")
Bump on this, I'm pretty sure that email tokenizing has been sorted since 16.0 ( [http://elasticsearch-users.115913.n3.nabble.com/Search-Email-Part-tp2835831p2835885.html](http://elasticsearch-users.115913.n3.nabble.com/Search-Email-Part-tp2835831p2835885.html) ) but for some reason I can't get it working. On Jan 9, 4:33 pm, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote: > Hi, > > I've defined an analyser and used it in a Mapping as defined below: > > [Email Tokenizer Problem · GitHub](https://gist.github.com/1583636) > > But when I enter something like @gmail.com, it is not matching the > wildcard search, it only matches when I enter "[test.u...@gmail.com](mailto:test.u...@gmail.com)" or > "test.user" or "test." etc. i.e. there appears to be no tokenization > on the "@" or the ".". > > This is the query I'm using: > > [Email Tokenizer Problem · GitHub](https://gist.github.com/1583708) > > Regards, > > David. ---
**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\ **Post date:** [January 11, 2012, 11:20am UTC](https://discuss.elastic.co/t/email-tokenizing-not-working/6329/3 "2012-01-11T11:20:35Z")
I've figured out what is happening here: The tokenizer uax\_url\_email treats the email as a single token, so what is implied in this forum post [http://elasticsearch-users.115913.n3.nabble.com/Search-Email-Part-td2835831.html](http://elasticsearch-users.115913.n3.nabble.com/Search-Email-Part-td2835831.html), i.e. that uax\_url\_email tokenizes the email is not correct. The explanation here is more accurate: [http://elasticsearch-users.115913.n3.nabble.com/Problem-with-searching-Emails-td2902363.html#a2902602](http://elasticsearch-users.115913.n3.nabble.com/Problem-with-searching-Emails-td2902363.html#a2902602). If you want to tokenize the email on "@" and "." then a pattern tokenizer can be used. This is an example mapping that has both: index: mapper: dynamic: false analysis: analyzer: default: type: standard stopwords: _none_ # sortable is used to enable case-independent sorting ``` sortable: tokenizer: keyword filter: [lowercase] ``` # uax\_url\_email is used to analyse emails as a single token. ``` uax_url_email: tokenizer: uax_url_email filter: [standard, lowercase, stop] ``` # email\_tokenizer is designed to tokenize emails on "." and "@" ``` email_tokenizer: type: pattern pattern: "[\\.@]" ``` Interestingly enough, its possible to get some quite nice partial matches using the uax\_url\_email tokenizer by creating a String query that allows leading wildcards, so for an email "test.user[gmail.com](http://gmail.com)", the following would all match using and leading and following wildcards (NB. use of leading wildcards is resource intensive and can affect performance, if used across more than one or two fields): "usergmail", "test." "@gmail" "er@gm" etc. -David. On Jan 10, 9:46 am, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote: > Bump on this, I'm pretty sure that email tokenizing has been sorted > since 16.0 ([http://elasticsearch-users.115913.n3.nabble.com/Search-Email-Part-tp2](http://elasticsearch-users.115913.n3.nabble.com/Search-Email-Part-tp2)... > ) but for some reason I can't get it working. > > On Jan 9, 4:33 pm, davrob2 [davirobe...@gmail.com](mailto:davirobe...@gmail.com) wrote: > > > Hi, > > > I've defined an analyser and used it in a Mapping as defined below: > > > [Email Tokenizer Problem · GitHub](https://gist.github.com/1583636) > > > But when I enter something like @gmail.com, it is not matching the > > wildcard search, it only matches when I enter "[test.u...@gmail.com](mailto:test.u...@gmail.com)" or > > "test.user" or "test." etc. i.e. there appears to be no tokenization > > on the "@" or the ".". > > > This is the query I'm using: > > > [Email Tokenizer Problem · GitHub](https://gist.github.com/1583708) > > > Regards, > > > David. ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [January 11, 2012, 11:43am UTC](https://discuss.elastic.co/t/email-tokenizing-not-working/6329/4 "2012-01-11T11:43:48Z")
> The tokenizer uax\_url\_email treats the email as a single token, so > what is implied in this forum post Ooh - interesting. I assumed that this sentence: ``` A tokenizer of type uax_url_email which works exactly like the standard tokenizer, but also handles emails and urls. ``` ...meant that it tokenized emails and urls into individual parts, not as single tokens. thanks for the post. I've updated the docs to reflect that: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. clint ---
**Author:** ![Ask\_Bjorn\_Hansen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ask_bjorn_hansen/32/2995_2.png) [@Ask\_Bjorn\_Hansen](https://discuss.elastic.co/u/Ask_Bjorn_Hansen)\ **Post date:** [February 14, 2012, 6:06am UTC](https://discuss.elastic.co/t/email-tokenizing-not-working/6329/5 "2012-02-14T06:06:44Z")
On Jan 11, 3:43 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote: > > Thetokenizeruax\_url\_email treats theemailas a single token, so > > what is implied in this forum post > > Ooh - interesting. I assumed that this sentence: > > ``` > Atokenizerof type uax_url_email which works exactly like the > standardtokenizer, but also handles emails and urls. > > ``` > > ...meant that it tokenized emails and urls into individual parts, not as > single tokens. On Jan 11, 3:43 am, Clinton Gormley [cl...@traveljury.com](mailto:cl...@traveljury.com) wrote: > > Thetokenizeruax\_url\_email treats theemailas a single token, so > > what is implied in this forum post > > Ooh - interesting. I assumed that this sentence: > > ``` > Atokenizerof type uax_url_email which works exactly like the > standardtokenizer, but also handles emails and urls. > > ``` > > ...meant that it tokenized emails and urls into individual parts, not as > single tokens. \_analyze seems to show something else, or am I calling it wrong? $ curl -XGET 'indexdev1.la.sol:9200/us-devel-rms-v1/\_analyze? tokenizer=uax\_url\_email&format=text' -d 'some.email@DOMAIN.COM'; echo {"tokens":"[some.email:0-\>10:]\n\n2: \n[domain.com:11- > 21:]\n"} Ask -- [http://askask.com/](http://askask.com/) ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [February 14, 2012, 9:46am UTC](https://discuss.elastic.co/t/email-tokenizing-not-working/6329/6 "2012-02-14T09:46:47Z")
> \_analyze seems to show something else, or am I calling it wrong? > > $ curl -XGET 'indexdev1.la.sol:9200/us-devel-rms-v1/\_analyze? > tokenizer=uax\_url\_email&format=text' -d 'some.email@DOMAIN.COM'; echo > {"tokens":"[some.email:0-\>10:]\n\n2: \n[domain.com:11- > > > 21:]\n"} What version of ES are you running? The form that allows you to pass 'tokenizer' params is only available as of 0.19.0RC1 Otherwise you're actually just using the 'standard' analyzer c ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:39am UTC](https://discuss.elastic.co/t/email-tokenizing-not-working/6329/7 "2017-07-06T03:39:29Z")