# ElasticSearch standard Analyzer - exceptional case

**URL:** <https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534>\
**Category:** Elasticsearch\
**Created:** [December 13, 2017, 10:23am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534 "2017-12-13T10:23:25Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![sanjaykanani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanjaykanani/32/25488_2.png) [@sanjaykanani](https://discuss.elastic.co/u/sanjaykanani)\
**Post date:** [December 13, 2017, 10:23am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/1 "2017-12-13T10:23:26Z")

</div>

Continuing the discussion from [ElasticSearch standard Analyzer - problematic case](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-problematic-case/16782):

[quote="Igor\_Romanov, post:1, topic:16782"]

Hi

I have read this article but no one has replied on it.  
I am facing same issue.please provide solution.

I was analyzing some analyzer weird behavior, and try to understand why it

happens and how to fix it

here what token I get for standard analyzer for text:

"myemail@email.com:test1234"

```
curl -XGET 'localhost:9200/_analyze?analyzer=standard&pretty=true' -d 

'myemail@email.com:test1234'

{

  "tokens" : [ {

    "token" : "myemail",

    "start_offset" : 0,

    "end_offset" : 7,

    "type" : "",

    "position" : 1

  }, {

    "token" : "email.com:test1234",

    "start_offset" : 8,

    "end_offset" : 26,

    "type" : "",

    "position" : 2

  } ]

}

```

so question is why I am getting that as one token: "[email.com](http://email.com):test1234"

why it is not divided to tokens by . : \_ and ?

Standard tokenizer is not working with this four keywords.

can any one help me out on this issue?  
Is it BUG or not?

Thanks,  
sanjay

---

<div class="post-metadata">

**Author:** ![Tom\_Mortimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_mortimer/32/21049_2.png) [@Tom\_Mortimer](https://discuss.elastic.co/u/Tom_Mortimer)\
**Post date:** [December 13, 2017, 10:41am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/2 "2017-12-13T10:41:33Z")

</div>

Hi Sanjay,

What do you _want_ to happen in this case? What are the search requirements? Can you give some examples of queries you would expect to match this email address in the text, and other which would you expect _not_ to match it?

Tom

---

<div class="post-metadata">

**Author:** ![sanjaykanani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanjaykanani/32/25488_2.png) [@sanjaykanani](https://discuss.elastic.co/u/sanjaykanani)\
**Post date:** [December 13, 2017, 10:52am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/3 "2017-12-13T10:52:45Z")

</div>

I just want to tokenize it with standard analyzer.  
when I add any text like "test.elastic wirth test:lucene" it should tokenized with dot and colon char as well as it's not doing.

so tokens are

test  
elastic  
lucene

---

<div class="post-metadata">

**Author:** ![Tom\_Mortimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_mortimer/32/21049_2.png) [@Tom\_Mortimer](https://discuss.elastic.co/u/Tom_Mortimer)\
**Post date:** [December 13, 2017, 10:59am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/4 "2017-12-13T10:59:04Z")

</div>

The Simple Analyzer does what you want, I think?

```auto
curl "localhost:9200/_analyze?analyzer=simple&text=myemail@email.com:test1234&pretty"
{
  "tokens" : [
    {
      "token" : "myemail",
      "start_offset" : 0,
      "end_offset" : 7,
      "type" : "word",
      "position" : 0
    },
    {
      "token" : "email",
      "start_offset" : 8,
      "end_offset" : 13,
      "type" : "word",
      "position" : 1
    },
    {
      "token" : "com",
      "start_offset" : 14,
      "end_offset" : 17,
      "type" : "word",
      "position" : 2
    },
    {
      "token" : "test",
      "start_offset" : 18,
      "end_offset" : 22,
      "type" : "word",
      "position" : 3
    }
  ]
}

```

---

<div class="post-metadata">

**Author:** ![sanjaykanani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanjaykanani/32/25488_2.png) [@sanjaykanani](https://discuss.elastic.co/u/sanjaykanani)\
**Post date:** [December 13, 2017, 11:03am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/5 "2017-12-13T11:03:12Z")

</div>

> [@Tom\_Mortimer](#):
>
> localhost:9200/\_analyze?analyzer=simple&text=myemail@email.com:test1234&pretty

Yes simple analyzer is one only for english language it not working in other language like gujarati,hindi,urdu,french etc..

so I can not remove standard analyzer I just want to tokenize with those char

thanks,

---

<div class="post-metadata">

**Author:** ![Tom\_Mortimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_mortimer/32/21049_2.png) [@Tom\_Mortimer](https://discuss.elastic.co/u/Tom_Mortimer)\
**Post date:** [December 13, 2017, 11:04am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/6 "2017-12-13T11:04:31Z")

</div>

Have you tried using a regular expression?

[https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-analyzer.html](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/analysis-pattern-analyzer.html)

---

<div class="post-metadata">

**Author:** ![sanjaykanani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanjaykanani/32/25488_2.png) [@sanjaykanani](https://discuss.elastic.co/u/sanjaykanani)\
**Post date:** [December 13, 2017, 11:08am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/7 "2017-12-13T11:08:05Z")

</div>

Not Yet just finding reason for this.  
do you know why this is happening with this 4 chars.

---

<div class="post-metadata">

**Author:** ![Tom\_Mortimer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tom_mortimer/32/21049_2.png) [@Tom\_Mortimer](https://discuss.elastic.co/u/Tom_Mortimer)\
**Post date:** [December 13, 2017, 11:18am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/8 "2017-12-13T11:18:21Z")

</div>

I guess because the designers of the standard analyzer decided to preserve e.g. domain names as single terms?

Text analysis in the real world is often a compromise and should be driven by your search requirements.

---

<div class="post-metadata">

**Author:** ![sanjaykanani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanjaykanani/32/25488_2.png) [@sanjaykanani](https://discuss.elastic.co/u/sanjaykanani)\
**Post date:** [December 13, 2017, 11:21am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/9 "2017-12-13T11:21:52Z")

</div>

no It's not working if I insert text "something.is.missing" .  
It will take it as whole word.  
It's not breaking word.

---

<div class="post-metadata">

**Author:** ![sanjaykanani](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sanjaykanani/32/25488_2.png) [@sanjaykanani](https://discuss.elastic.co/u/sanjaykanani)\
**Post date:** [December 13, 2017, 11:40am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/10 "2017-12-13T11:40:05Z")

</div>

hi,  
Please check below link .  
Same solution we need in elastic search.  
[https://stackoverflow.com/questions/15235126/lucene-4-1-how-split-words-that-contains-dots-when-indexing](https://stackoverflow.com/questions/15235126/lucene-4-1-how-split-words-that-contains-dots-when-indexing)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 10, 2018, 11:40am UTC](https://discuss.elastic.co/t/elasticsearch-standard-analyzer-exceptional-case/111534/11 "2018-01-10T11:40:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.