# Searching for "foo" should also find occurrence of "foo.bar"

**URL:** <https://discuss.elastic.co/t/searching-for-foo-should-also-find-occurrence-of-foo-bar/23720>\
**Category:** Elasticsearch\
**Created:** [May 29, 2015, 7:39am UTC](https://discuss.elastic.co/t/searching-for-foo-should-also-find-occurrence-of-foo-bar/23720 "2015-05-29T07:39:17Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![marian](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marian/32/442_2.png) [@marian](https://discuss.elastic.co/u/marian)\
**Post date:** [May 29, 2015, 7:39am UTC](https://discuss.elastic.co/t/searching-for-foo-should-also-find-occurrence-of-foo-bar/23720/1 "2015-05-29T07:39:17Z")

</div>

We have ElasticSearch 1.5 set up with a very simple mapping to perform full  
text search in our docs ([https://docs.giantswarm.io/](https://docs.giantswarm.io/)). When searching for  
"swarmvars" we get no hits, although "swarmvars.json" appears in documents.

The field "text" is used as a catch-all field for all searchable content  
(title, document body, keywords). Here is the mapping:

"properties": {  
...,  
"text": {  
"type": "string",  
"store": true,  
"index": "analyzed",  
"term\_vector": "with\_positions\_offsets",  
"analyzer": "english",  
}  
}

When using the "english" analyzer on the text "Text containing  
swarmvars.json and more", the result are these tokens:

text  
contain  
swarmvars.json  
more

Having the token "swarmvars.json" is fine. What I need are two additional  
tokens "swarmvars" and "json". How can I achieve that?

I was looking into creating a custom tokenizer, but I was unable to get it  
to work (errors when applying the settings) and also I was unable to find  
an example, no matter how I searched.

Thanks!

## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/85a03096-ae33-4517-8eab-6f2be4da73ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85a03096-ae33-4517-8eab-6f2be4da73ed%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 29, 2015, 8:47am UTC](https://discuss.elastic.co/t/searching-for-foo-should-also-find-occurrence-of-foo-bar/23720/2 "2015-05-29T08:47:45Z")

</div>

I would probably go with a Pattern Tokenizer and define whatever regex you need.  
[Pattern tokenizer | Reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html) [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html)

The standard one is more for english text which means that a dot need to have a space after it in order to be considered as a break between two tokens.

Make sense?

--  
David Pilato - Developer | Evangelist

> **[Elastic — The Search AI Company](https://www.elastic.co/)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...

@dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)

> Le 29 mai 2015 à 09:39, Marian Steinbach [marian.steinbach@gmail.com](mailto:marian.steinbach@gmail.com) a écrit :
> 
> We have Elasticsearch 1.5 set up with a very simple mapping to perform full text search in our docs ([https://docs.giantswarm.io/](https://docs.giantswarm.io/)). When searching for "swarmvars" we get no hits, although "swarmvars.json" appears in documents.
> 
> The field "text" is used as a catch-all field for all searchable content (title, document body, keywords). Here is the mapping:
> 
> "properties": {  
> ...,  
> "text": {  
> "type": "string",  
> "store": true,  
> "index": "analyzed",  
> "term\_vector": "with\_positions\_offsets",  
> "analyzer": "english",  
> }  
> }
> 
> When using the "english" analyzer on the text "Text containing swarmvars.json and more", the result are these tokens:
> 
> text  
> contain  
> swarmvars.json  
> more
> 
> Having the token "swarmvars.json" is fine. What I need are two additional tokens "swarmvars" and "json". How can I achieve that?
> 
> I was looking into creating a custom tokenizer, but I was unable to get it to work (errors when applying the settings) and also I was unable to find an example, no matter how I searched.
> 
> Thanks!
> 
> ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) [https://discuss.elastic.co/](https://discuss.elastic.co/)
> 
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/85a03096-ae33-4517-8eab-6f2be4da73ed%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/85a03096-ae33-4517-8eab-6f2be4da73ed%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/85a03096-ae33-4517-8eab-6f2be4da73ed%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/85a03096-ae33-4517-8eab-6f2be4da73ed%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8B2107E8-4E9A-47FA-BFE0-FE36FF9FBF1C%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/8B2107E8-4E9A-47FA-BFE0-FE36FF9FBF1C%40pilato.fr).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![marian](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marian/32/442_2.png) [@marian](https://discuss.elastic.co/u/marian)\
**Post date:** [May 29, 2015, 8:59am UTC](https://discuss.elastic.co/t/searching-for-foo-should-also-find-occurrence-of-foo-bar/23720/3 "2015-05-29T08:59:46Z")

</div>

Thanks for the reply! However, it doesn't make sense to me directly.

If I use the dot as an additional seperator, I will end up with the tokens  
"swarmvars" and "json", but not "swarmvars.json". Right?

Am Freitag, 29. Mai 2015 10:47:56 UTC+2 schrieb David Pilato:

> I would probably go with a Pattern Tokenizer and define whatever regex you  
> need.
> 
> [Pattern tokenizer | Reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html)
> 
> The standard one is more for english text which means that a dot need to  
> have a space after it in order to be considered as a break between two  
> tokens.
> 
> Make sense?

## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/44d85c90-acad-43b9-a082-6343395f19c5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/44d85c90-acad-43b9-a082-6343395f19c5%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 29, 2015, 9:02am UTC](https://discuss.elastic.co/t/searching-for-foo-should-also-find-occurrence-of-foo-bar/23720/4 "2015-05-29T09:02:12Z")

</div>

Yes. Because « Hello. How are you? » is a sentence that can be broken in « hello », « how », « are », « you ».  
But in « I paid it 2.50 euros », I would most likely keep « 2.50 » as a whole token.

--  
David Pilato - Developer | Evangelist

> **[Elastic — The Search AI Company](https://www.elastic.co/)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...

@dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)

> Le 29 mai 2015 à 10:59, Marian Steinbach [marian.steinbach@gmail.com](mailto:marian.steinbach@gmail.com) a écrit :
> 
> Thanks for the reply! However, it doesn't make sense to me directly.
> 
> If I use the dot as an additional seperator, I will end up with the tokens "swarmvars" and "json", but not "swarmvars.json". Right?
> 
> Am Freitag, 29. Mai 2015 10:47:56 UTC+2 schrieb David Pilato:  
> I would probably go with a Pattern Tokenizer and define whatever regex you need.  
> [Pattern tokenizer | Reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html) [https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-pattern-tokenizer.html)
> 
> The standard one is more for english text which means that a dot need to have a space after it in order to be considered as a break between two tokens.
> 
> Make sense?
> 
> ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) [https://discuss.elastic.co/](https://discuss.elastic.co/)
> 
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/44d85c90-acad-43b9-a082-6343395f19c5%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/44d85c90-acad-43b9-a082-6343395f19c5%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/44d85c90-acad-43b9-a082-6343395f19c5%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/44d85c90-acad-43b9-a082-6343395f19c5%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/DD60ACAE-9659-43F1-AF10-6517D0D79DEF%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/DD60ACAE-9659-43F1-AF10-6517D0D79DEF%40pilato.fr).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![marian](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marian/32/442_2.png) [@marian](https://discuss.elastic.co/u/marian)\
**Post date:** [May 29, 2015, 9:11am UTC](https://discuss.elastic.co/t/searching-for-foo-should-also-find-occurrence-of-foo-bar/23720/5 "2015-05-29T09:11:39Z")

</div>

Am Freitag, 29. Mai 2015 11:02:25 UTC+2 schrieb David Pilato:

> Yes. Because « Hello. How are you? » is a sentence that can be broken in  
> « hello », « how », « are », « you ».  
> But in « I paid it 2.50 euros », I would most likely keep « 2.50 » as a  
> whole token.

So far, so easy. And my question is now: From a text "foo.bar", how can I  
generate ALL of the following tokens?

foo  
bar  
foo.bar

## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e7acf73b-1e15-431c-bee7-1b5f726fb69d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e7acf73b-1e15-431c-bee7-1b5f726fb69d%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [May 29, 2015, 9:13am UTC](https://discuss.elastic.co/t/searching-for-foo-should-also-find-occurrence-of-foo-bar/23720/6 "2015-05-29T09:13:50Z")

</div>

I would use 2 analyzers and multi field: [Field data types | Reference](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html#_multi_fields_3) [https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html#\_multi\_fields\_3](https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-core-types.html#_multi_fields_3)

--  
David Pilato - Developer | Evangelist

> **[Elastic — The Search AI Company](https://www.elastic.co/)**
>
> Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ...

@dadoonet [https://twitter.com/dadoonet](https://twitter.com/dadoonet) | @elasticsearchfr [https://twitter.com/elasticsearchfr](https://twitter.com/elasticsearchfr) | @scrutmydocs [https://twitter.com/scrutmydocs](https://twitter.com/scrutmydocs)

> Le 29 mai 2015 à 11:11, Marian Steinbach [marian.steinbach@gmail.com](mailto:marian.steinbach@gmail.com) a écrit :
> 
> Am Freitag, 29. Mai 2015 11:02:25 UTC+2 schrieb David Pilato:  
> Yes. Because « Hello. How are you? » is a sentence that can be broken in « hello », « how », « are », « you ».  
> But in « I paid it 2.50 euros », I would most likely keep « 2.50 » as a whole token.
> 
> So far, so easy. And my question is now: From a text "foo.bar", how can I generate ALL of the following tokens?
> 
> foo  
> bar  
> foo.bar
> 
> ## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/) [https://discuss.elastic.co/](https://discuss.elastic.co/)
> 
> You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com) [mailto:elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e7acf73b-1e15-431c-bee7-1b5f726fb69d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e7acf73b-1e15-431c-bee7-1b5f726fb69d%40googlegroups.com) [https://groups.google.com/d/msgid/elasticsearch/e7acf73b-1e15-431c-bee7-1b5f726fb69d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e7acf73b-1e15-431c-bee7-1b5f726fb69d%40googlegroups.com?utm_medium=email&utm_source=footer).  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout) [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

## -- Please update your bookmarks! We have moved to [https://discuss.elastic.co/](https://discuss.elastic.co/)

You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8F8EF922-9156-4DD9-98DA-1D1B5ECF3929%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/8F8EF922-9156-4DD9-98DA-1D1B5ECF3929%40pilato.fr).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:10am UTC](https://discuss.elastic.co/t/searching-for-foo-should-also-find-occurrence-of-foo-bar/23720/7 "2017-07-06T00:10:52Z")

</div>