# Problem when using analyzers (very small data set)

**URL:** <https://discuss.elastic.co/t/problem-when-using-analyzers-very-small-data-set/10999>\
**Category:** Elasticsearch\
**Created:** [March 4, 2013, 3:42pm UTC](https://discuss.elastic.co/t/problem-when-using-analyzers-very-small-data-set/10999 "2013-03-04T15:42:31Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul\_Walsh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_walsh/32/2447_2.png) [@Paul\_Walsh](https://discuss.elastic.co/u/Paul_Walsh)\
**Post date:** [March 4, 2013, 3:42pm UTC](https://discuss.elastic.co/t/problem-when-using-analyzers-very-small-data-set/10999/1 "2013-03-04T15:42:31Z")

</div>

Hi,

I've been trying to use some new analyzers in my ES instance (version  
0.20.4) and I've been noticing some problems on search. I've been trying to  
follow this simple  
example: [http://mnylen.tumblr.com/post/22963609412/elasticsearch-and-a-simple-contains-search](http://mnylen.tumblr.com/post/22963609412/elasticsearch-and-a-simple-contains-search)

Here's how I set up my data:

curl -XPUT "[http://localhost:9200/catalog/?pretty](http://localhost:9200/catalog/?pretty)" -d '{

"mappings" : {

```
"product" : {

  "properties" : {

    "title" : {

      "type" : "string",

      "search_analyzer" : "str_search_analyzer",

      "index_analyzer" : "str_index_analyzer"

    }

  }

}

```

},

"settings" : {

```
"analysis" : {

  "analyzer" : {

    "str_search_analyzer" : {

      *"tokenizer" : "whitespace",*

      "filter" : ["lowercase"]

    },

    "str_index_analyzer" : {

      *"tokenizer" : "whitespace",*

      "filter" : ["lowercase", "substring"]

    }

  },

  "filter" : {

    "substring" : {

      "type" : "nGram",

      *"min_gram"** : 6*,

      "max_gram" : 20

    }

  }

}

```

}

}';

curl -XPOST "[http://localhost:9200/catalog/product?pretty](http://localhost:9200/catalog/product?pretty)" -d '{

"title" : "Logitech Wireless Keyboard K350"

}';

curl -XPOST "[http://localhost:9200/catalog/product?pretty](http://localhost:9200/catalog/product?pretty)" -d '{

"title" : "Das Keyboard"

}'

Some things I changed from the example where _"tokenizer" : "keyword"_ is  
now _"tokenizer" : "whitespace"_ and _"min\_gram"_ is now 6 instead of 1.

The first problem is that unless I specify a field, searching with tokens  
don't work.

The tokens for the word "Logitech" are "gitech", "logite", "logitec",  
"logitech", "ogitec" and "ogitech". So I would expect that searching any of  
those substrings would return me the Logitech result, however this returns  
no results:

[http://localhost:9200/catalog/\_search?q=ogitec](http://localhost:9200/catalog/_search?q=ogitec)  
{"took": 9,"timed\_out": false,"\_shards": {"total": 5,"successful": 5,"failed  
": 0},"hits": {"total": 0,"max\_score": null,"hits": []}}

If I do the same search and specify title, only then do I get the expected  
result:

[http://localhost:9200/catalog/\_search?q=title:ogitec](http://localhost:9200/catalog/_search?q=title:ogitec)  
{"took": 4,"timed\_out": false,"\_shards": {"total": 5,"successful": 5,"failed  
": 0},"hits": {"total": 1,"max\_score": 0.067124054,"hits": [{"\_index": "  
catalog","\_type": "product","\_id": "J4trtjx9Rwm5jdI0kqmygA","\_score":  
0.067124054,"\_source": {"title": "Logitech Wireless Keyboard K350"}}]}}

Does anyone know why this is happening?

The second thing I noticed (using the same data setup) was that when I  
analyze the strings I'm searching I get two different types. Using the  
default analyzer on the index catalog shows the type as an :

[http://localhost:9200/catalog/\_analyze?pretty&text=ogitec](http://localhost:9200/catalog/_analyze?pretty&text=ogitec)  
{"tokens": [{"token": "ogitec","start\_offset": 0,"end\_offset": 6,"type": "  
","position": 1}]}  
But using my custom analyzer shows the type as a "word":

[http://localhost:9200/catalog/\_analyze?pretty&text=ogitec&analyzer=str\_search\_analyzer](http://localhost:9200/catalog/_analyze?pretty&text=ogitec&analyzer=str_search_analyzer)  
{"tokens": [{"token": "ogitec","start\_offset": 0,"end\_offset": 6,"type": "  
word","position": 1}]}

I'm wondering if this has something to do with the problem above  
(searching over all fields in an index as opposed to a specified one), and  
why it is happening.

Any help is appreciated.  
(Also, sorry for the length of the post. I didn't want to leave out any  
information)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![egaumer](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/egaumer/32/2365_2.png) [@egaumer](https://discuss.elastic.co/u/egaumer)\
**Post date:** [March 4, 2013, 5:39pm UTC](https://discuss.elastic.co/t/problem-when-using-analyzers-very-small-data-set/10999/2 "2013-03-04T17:39:05Z")

</div>

You're only applying this custom analysis on the title field in your  
mapping properties. If you don't specify that field in your searches then a  
default field is used (\_all in this case).

-Eric

On Monday, March 4, 2013 10:42:31 AM UTC-5, Paul W wrote:

> Hi,
> 
> I've been trying to use some new analyzers in my ES instance (version  
> 0.20.4) and I've been noticing some problems on search. I've been trying to  
> follow this simple example:  
> [http://mnylen.tumblr.com/post/22963609412/elasticsearch-and-a-simple-contains-search](http://mnylen.tumblr.com/post/22963609412/elasticsearch-and-a-simple-contains-search)
> 
> Here's how I set up my data:
> 
> curl -XPUT "[http://localhost:9200/catalog/?pretty](http://localhost:9200/catalog/?pretty)" -d '{
> 
> "mappings" : {
> 
> ```
> "product" : {
> 
> "properties" : {
> 
> "title" : {
> 
> "type" : "string",
> 
> "search_analyzer" : "str_search_analyzer",
> 
> "index_analyzer" : "str_index_analyzer"
> 
> }
> 
> }
> 
> }
> 
> ```
> 
> },
> 
> "settings" : {
> 
> ```
> "analysis" : {
> 
> "analyzer" : {
> 
> "str_search_analyzer" : {
> 
> *"tokenizer" : "whitespace",*
> 
> "filter" : ["lowercase"]
> 
> },
> 
> "str_index_analyzer" : {
> 
> *"tokenizer" : "whitespace",*
> 
> "filter" : ["lowercase", "substring"]
> 
> }
> 
> },
> 
> "filter" : {
> 
> "substring" : {
> 
> "type" : "nGram",
> 
> *"min_gram"** : 6*,
> 
> "max_gram" : 20
> 
> }
> 
> }
> 
> }
> 
> ```
> 
> }
> 
> }';
> 
> curl -XPOST "[http://localhost:9200/catalog/product?pretty](http://localhost:9200/catalog/product?pretty)" -d '{
> 
> "title" : "Logitech Wireless Keyboard K350"
> 
> }';
> 
> curl -XPOST "[http://localhost:9200/catalog/product?pretty](http://localhost:9200/catalog/product?pretty)" -d '{
> 
> "title" : "Das Keyboard"
> 
> }'
> 
> Some things I changed from the example where _"tokenizer" : "keyword"_ is  
> now _"tokenizer" : "whitespace"_ and _"min\_gram"_ is now 6 instead of 1.
> 
> The first problem is that unless I specify a field, searching with tokens  
> don't work.
> 
> The tokens for the word "Logitech" are "gitech", "logite", "logitec",  
> "logitech", "ogitec" and "ogitech". So I would expect that searching any of  
> those substrings would return me the Logitech result, however this returns  
> no results:
> 
> [http://localhost:9200/catalog/\_search?q=ogitec](http://localhost:9200/catalog/_search?q=ogitec)  
> {"took": 9,"timed\_out": false,"\_shards": {"total": 5,"successful": 5,"  
> failed": 0},"hits": {"total": 0,"max\_score": null,"hits": }}
> 
> If I do the same search and specify title, only then do I get the expected  
> result:
> 
> [http://localhost:9200/catalog/\_search?q=title:ogitec](http://localhost:9200/catalog/_search?q=title:ogitec)  
> {"took": 4,"timed\_out": false,"\_shards": {"total": 5,"successful": 5,"  
> failed": 0},"hits": {"total": 1,"max\_score": 0.067124054,"hits": [{"\_index  
> ": "catalog","\_type": "product","\_id": "J4trtjx9Rwm5jdI0kqmygA","\_score":  
> 0.067124054,"\_source": {"title": "Logitech Wireless Keyboard K350"}}]}}
> 
> Does anyone know why this is happening?
> 
> The second thing I noticed (using the same data setup) was that when I  
> analyze the strings I'm searching I get two different types. Using the  
> default analyzer on the index catalog shows the type as an :
> 
> [http://localhost:9200/catalog/\_analyze?pretty&text=ogitec](http://localhost:9200/catalog/_analyze?pretty&text=ogitec)  
> {"tokens": [{"token": "ogitec","start\_offset": 0,"end\_offset": 6,"type": "  
> ","position": 1}]}  
> But using my custom analyzer shows the type as a "word":
> 
> [http://localhost:9200/catalog/\_analyze?pretty&text=ogitec&analyzer=str\_search\_analyzer](http://localhost:9200/catalog/_analyze?pretty&text=ogitec&analyzer=str_search_analyzer)  
> {"tokens": [{"token": "ogitec","start\_offset": 0,"end\_offset": 6,"type": "  
> word","position": 1}]}
> 
> I'm wondering if this has something to do with the problem above  
> (searching over all fields in an index as opposed to a specified one), and  
> why it is happening.
> 
> Any help is appreciated.  
> (Also, sorry for the length of the post. I didn't want to leave out any  
> information)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Paul\_Walsh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_walsh/32/2447_2.png) [@Paul\_Walsh](https://discuss.elastic.co/u/Paul_Walsh)\
**Post date:** [March 5, 2013, 10:08am UTC](https://discuss.elastic.co/t/problem-when-using-analyzers-very-small-data-set/10999/3 "2013-03-05T10:08:45Z")

</div>

Oh ok. Well that makes sense. Cool, thanks.

Also, do you have any idea why the type is showing as "" with  
standard analysis and "word" when I use my custom analyzer? Do you think  
this will make a difference when searching?

On Monday, 4 March 2013 17:39:05 UTC, egaumer wrote:

> You're only applying this custom analysis on the title field in your  
> mapping properties. If you don't specify that field in your searches then a  
> default field is used (\_all in this case).
> 
> -Eric
> 
> On Monday, March 4, 2013 10:42:31 AM UTC-5, Paul W wrote:
> 
> > Hi,
> > 
> > I've been trying to use some new analyzers in my ES instance (version  
> > 0.20.4) and I've been noticing some problems on search. I've been trying to  
> > follow this simple example:  
> > [http://mnylen.tumblr.com/post/22963609412/elasticsearch-and-a-simple-contains-search](http://mnylen.tumblr.com/post/22963609412/elasticsearch-and-a-simple-contains-search)
> > 
> > Here's how I set up my data:
> > 
> > curl -XPUT "[http://localhost:9200/catalog/?pretty](http://localhost:9200/catalog/?pretty)" -d '{
> > 
> > "mappings" : {
> > 
> > ```
> > "product" : {
> > 
> > "properties" : {
> > 
> > "title" : {
> > 
> > "type" : "string",
> > 
> > "search_analyzer" : "str_search_analyzer",
> > 
> > "index_analyzer" : "str_index_analyzer"
> > 
> > }
> > 
> > }
> > 
> > }
> > 
> > ```
> > 
> > },
> > 
> > "settings" : {
> > 
> > ```
> > "analysis" : {
> > 
> > "analyzer" : {
> > 
> > "str_search_analyzer" : {
> > 
> > *"tokenizer" : "whitespace",*
> > 
> > "filter" : ["lowercase"]
> > 
> > },
> > 
> > "str_index_analyzer" : {
> > 
> > *"tokenizer" : "whitespace",*
> > 
> > "filter" : ["lowercase", "substring"]
> > 
> > }
> > 
> > },
> > 
> > "filter" : {
> > 
> > "substring" : {
> > 
> > "type" : "nGram",
> > 
> > *"min_gram"** : 6*,
> > 
> > "max_gram" : 20
> > 
> > }
> > 
> > }
> > 
> > }
> > 
> > ```
> > 
> > }
> > 
> > }';
> > 
> > curl -XPOST "[http://localhost:9200/catalog/product?pretty](http://localhost:9200/catalog/product?pretty)" -d '{
> > 
> > "title" : "Logitech Wireless Keyboard K350"
> > 
> > }';
> > 
> > curl -XPOST "[http://localhost:9200/catalog/product?pretty](http://localhost:9200/catalog/product?pretty)" -d '{
> > 
> > "title" : "Das Keyboard"
> > 
> > }'
> > 
> > Some things I changed from the example where _"tokenizer" : "keyword"_ is  
> > now _"tokenizer" : "whitespace"_ and _"min\_gram"_ is now 6 instead of 1.
> > 
> > The first problem is that unless I specify a field, searching with tokens  
> > don't work.
> > 
> > The tokens for the word "Logitech" are "gitech", "logite", "logitec",  
> > "logitech", "ogitec" and "ogitech". So I would expect that searching any of  
> > those substrings would return me the Logitech result, however this returns  
> > no results:
> > 
> > [http://localhost:9200/catalog/\_search?q=ogitec](http://localhost:9200/catalog/_search?q=ogitec)  
> > {"took": 9,"timed\_out": false,"\_shards": {"total": 5,"successful": 5,"  
> > failed": 0},"hits": {"total": 0,"max\_score": null,"hits": }}
> > 
> > If I do the same search and specify title, only then do I get the  
> > expected result:
> > 
> > [http://localhost:9200/catalog/\_search?q=title:ogitec](http://localhost:9200/catalog/_search?q=title:ogitec)  
> > {"took": 4,"timed\_out": false,"\_shards": {"total": 5,"successful": 5,"  
> > failed": 0},"hits": {"total": 1,"max\_score": 0.067124054,"hits": [{"  
> > \_index": "catalog","\_type": "product","\_id": "J4trtjx9Rwm5jdI0kqmygA","  
> > \_score": 0.067124054,"\_source": {"title": "Logitech Wireless Keyboard  
> > K350"}}]}}
> > 
> > Does anyone know why this is happening?
> > 
> > The second thing I noticed (using the same data setup) was that when I  
> > analyze the strings I'm searching I get two different types. Using the  
> > default analyzer on the index catalog shows the type as an :
> > 
> > [http://localhost:9200/catalog/\_analyze?pretty&text=ogitec](http://localhost:9200/catalog/_analyze?pretty&text=ogitec)  
> > {"tokens": [{"token": "ogitec","start\_offset": 0,"end\_offset": 6,"type":  
> > "","position": 1}]}  
> > But using my custom analyzer shows the type as a "word":
> > 
> > [http://localhost:9200/catalog/\_analyze?pretty&text=ogitec&analyzer=str\_search\_analyzer](http://localhost:9200/catalog/_analyze?pretty&text=ogitec&analyzer=str_search_analyzer)  
> > {"tokens": [{"token": "ogitec","start\_offset": 0,"end\_offset": 6,"type":  
> > "word","position": 1}]}
> > 
> > I'm wondering if this has something to do with the problem above  
> > (searching over all fields in an index as opposed to a specified one), and  
> > why it is happening.
> > 
> > Any help is appreciated.  
> > (Also, sorry for the length of the post. I didn't want to leave out any  
> > information)

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:48am UTC](https://discuss.elastic.co/t/problem-when-using-analyzers-very-small-data-set/10999/4 "2017-07-06T02:48:20Z")

</div>