# Query string not working with keyword tokenizer **URL:** \ **Category:** Elasticsearch\ **Created:** [January 14, 2011, 9:48pm UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778 "2011-01-14T21:48:59Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Andrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrei/32/2856_2.png) [@Andrei](https://discuss.elastic.co/u/Andrei)\ **Post date:** [January 14, 2011, 9:48pm UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/1 "2011-01-14T21:48:59Z")
I have a custom analyzer that looks like this: eulang\_tags: type: custom tokenizer: keyword filter: [lowercase, asciifolding] Then in my mapping, I set the "tags" field to use it like this: "tags" : {"type" : "string", "index\_name" : "tag", "boost" : 1.5, "analyzer" : "eulang\_tags"} When I try to do a query\_string type query against the tags field, it doesn't seem to work. For example, I have a document that contains a tag "coney island". If I issue this query: { "query\_string" : { "fields" : ["tags"], "query" : "coney island", "use\_dis\_max" : true } } The document is not returned. However, if I switch to "term" query: { "term" : { "tags": "coney island" } } The document is found successfully. Why is this happening? ---
**Author:** ![Adriano\_Ferreira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriano_ferreira/32/3228_2.png) [@Adriano\_Ferreira](https://discuss.elastic.co/u/Adriano_Ferreira)\ **Post date:** [January 14, 2011, 10:40pm UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/2 "2011-01-14T22:40:01Z")
On Fri, Jan 14, 2011 at 7:48 PM, Andrei [andrei@zmievski.org](mailto:andrei@zmievski.org) wrote: > I have a custom analyzer that looks like this: > > eulang\_tags: > type: custom > tokenizer: keyword > filter: [lowercase, asciifolding] > > Then in my mapping, I set the "tags" field to use it like this: > > "tags" : {"type" : "string", "index\_name" : "tag", "boost" : 1.5, > "analyzer" : "eulang\_tags"} > > When I try to do a query\_string type query against the tags field, it > doesn't seem to work. For example, I have a document that contains a > tag "coney island". If I issue this query: > > { > "query\_string" : { > "fields" : ["tags"], > "query" : "coney island", > "use\_dis\_max" : true > } > } This is using the "standard" analyzer, which breaks the "coney island" into terms "coney" and "island". In your doc which uses a "keyword" analyzer, the term in there is "coney island" itself and not the words. So use the "term" query like you did below, or explicitly tell the "query\_string" which analyzer you want to use (but I am not certain "keyword" and "query\_string" parse will work well together). > The document is not returned. However, if I switch to "term" query: > > { > "term" : { > "tags": "coney island" > } > } > > The document is found successfully. Why is this happening? ---
**Author:** ![Andrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrei/32/2856_2.png) [@Andrei](https://discuss.elastic.co/u/Andrei)\ **Post date:** [January 15, 2011, 1:28am UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/3 "2011-01-15T01:28:59Z")
I am not sure where you see the "standard" analyzer. I created the "eulang\_tags" custom analyzer as shown above and set the "tags" field to use it. -Andrei On Jan 14, 2:40 pm, Adriano Ferreira [a.r.ferre...@gmail.com](mailto:a.r.ferre...@gmail.com) wrote: > On Fri, Jan 14, 2011 at 7:48 PM, Andrei [and...@zmievski.org](mailto:and...@zmievski.org) wrote: > > > I have a custom analyzer that looks like this: > > > eulang\_tags: > > type: custom > > tokenizer: keyword > > filter: [lowercase, asciifolding] > > > Then in my mapping, I set the "tags" field to use it like this: > > > "tags" : {"type" : "string", "index\_name" : "tag", "boost" : 1.5, > > "analyzer" : "eulang\_tags"} > > > When I try to do a query\_string type query against the tags field, it > > doesn't seem to work. For example, I have a document that contains a > > tag "coney island". If I issue this query: > > > { > > "query\_string" : { > > "fields" : ["tags"], > > "query" : "coney island", > > "use\_dis\_max" : true > > } > > } > > This is using the "standard" analyzer, which breaks the "coney island" into > terms "coney" and "island". In your doc which uses a "keyword" analyzer, the > term in there is "coney island" itself and not the words. So use the "term" > query like you did below, or explicitly tell the "query\_string" which > analyzer you want to use (but I am not certain "keyword" and "query\_string" > parse will work well together). > > > The document is not returned. However, if I switch to "term" query: > > > { > > "term" : { > > "tags": "coney island" > > } > > } > > > The document is found successfully. Why is this happening? ---
**Author:** ![Adriano\_Ferreira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adriano_ferreira/32/3228_2.png) [@Adriano\_Ferreira](https://discuss.elastic.co/u/Adriano_Ferreira)\ **Post date:** [January 15, 2011, 4:09pm UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/4 "2011-01-15T16:09:53Z")
On Fri, Jan 14, 2011 at 11:28 PM, Andrei [andrei@zmievski.org](mailto:andrei@zmievski.org) wrote: > I am not sure where you see the "standard" analyzer. I created the > "eulang\_tags" custom analyzer as shown above and set the "tags" field > to use it. The "standard" analyzer should be the default analyzer for "query\_string" unless you change the defaults or explicitly request another analyzer. This is not tied to the fact you have used a custom analyzer to some of your fields in a certain type and index. > -Andrei > > On Jan 14, 2:40 pm, Adriano Ferreira [a.r.ferre...@gmail.com](mailto:a.r.ferre...@gmail.com) wrote: > > > On Fri, Jan 14, 2011 at 7:48 PM, Andrei [and...@zmievski.org](mailto:and...@zmievski.org) wrote: > > > > > I have a custom analyzer that looks like this: > > > > > eulang\_tags: > > > type: custom > > > tokenizer: keyword > > > filter: [lowercase, asciifolding] > > > > > Then in my mapping, I set the "tags" field to use it like this: > > > > > "tags" : {"type" : "string", "index\_name" : "tag", "boost" : 1.5, > > > "analyzer" : "eulang\_tags"} > > > > > When I try to do a query\_string type query against the tags field, it > > > doesn't seem to work. For example, I have a document that contains a > > > tag "coney island". If I issue this query: > > > > > { > > > "query\_string" : { > > > "fields" : ["tags"], > > > "query" : "coney island", > > > "use\_dis\_max" : true > > > } > > > } > > > > This is using the "standard" analyzer, which breaks the "coney island" > > into > > terms "coney" and "island". In your doc which uses a "keyword" analyzer, > > the > > term in there is "coney island" itself and not the words. So use the > > "term" > > query like you did below, or explicitly tell the "query\_string" which > > analyzer you want to use (but I am not certain "keyword" and > > "query\_string" > > parse will work well together). > > > > > The document is not returned. However, if I switch to "term" query: > > > > > { > > > "term" : { > > > "tags": "coney island" > > > } > > > } > > > > > The document is found successfully. Why is this happening? ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [January 15, 2011, 4:30pm UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/5 "2011-01-15T16:30:08Z")
> > ``` > > I am not sure where you see the "standard" analyzer. I > > created the > > "eulang_tags" custom analyzer as shown above and set the > > "tags" field > > to use it. > > > > ``` > The "standard" analyzer should be the default analyzer for > "query\_string" unless you change the defaults or explicitly request > another analyzer. This is not tied to the fact you have used a custom > analyzer to some of your fields in a certain type and index. Actually, the ES docs for mapping mention: - index\_analyzer: used when indexing a field - search\_analyzer: used when analyzing a field that is part of a query string - analyzer: sets both index\_analyzer and search\_analyzer See [Field data types | Elasticsearch Guide [8.11] | Elastic](http://www.elasticsearch.com/docs/elasticsearch/mapping/core_types/) So, my reading of this is that it should work - Andrei is, after all, searching on "fields": ["tags"] This may be a bug. Andrei, what happens if you search for: - "query\_string": "tags:coney\ island" - "query\_string": "tags:coney\ island" - "query\_string": "tags:(coney island)" I'm suggesting a few possibilities, because I'm not sure how to represent an embedded space in the query string. clint ---
**Author:** ![Andrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrei/32/2856_2.png) [@Andrei](https://discuss.elastic.co/u/Andrei)\ **Post date:** [January 15, 2011, 10:54pm UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/6 "2011-01-15T22:54:18Z")
Clint, "query\_string": "tags:coney\ island" seemed to work. Though now I'm not sure whether I should just escape the space this way or other non- alphanumeric characters as well. Maybe Shay can shed some light on this. -Andrei On Jan 15, 8:30 am, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote: > Actually, the ES docs for mapping mention: > > - index\_analyzer: used when indexing a field > - search\_analyzer: used when analyzing a field that is part of > a query string > - analyzer: sets both index\_analyzer and search\_analyzer > > Seehttp://www.elasticsearch.com/docs/elasticsearch/mapping/core\_types/ > > So, my reading of this is that it should work - Andrei is, after all, > searching on "fields": ["tags"] > > This may be a bug. > > Andrei, what happens if you search for: > > - "query\_string": "tags:coney\ island" > - "query\_string": "tags:coney\ island" > - "query\_string": "tags:(coney island)" > > I'm suggesting a few possibilities, because I'm not sure how to > represent an embedded space in the query string. > > clint ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 16, 2011, 9:55am UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/7 "2011-01-16T09:55:45Z")
The query parser breaks down on whitespaces as well. So, the things that gets passed to the analyzer and then construct the query is "coney" and then "island" without doing the actual escaping. Not ideal, but thats how it works... On Sunday, January 16, 2011 at 12:54 AM, Andrei wrote: > Clint, > > "query\_string": "tags:coney\ island" seemed to work. Though now I'm > not sure whether I should just escape the space this way or other non- > alphanumeric characters as well. Maybe Shay can shed some light on > this. > > -Andrei > > On Jan 15, 8:30 am, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote: > > > Actually, the ES docs for mapping mention: > > > > - index\_analyzer: used when indexing a field > > - search\_analyzer: used when analyzing a field that is part of > > a query string > > - analyzer: sets both index\_analyzer and search\_analyzer > > > > Seehttp://www.elasticsearch.com/docs/elasticsearch/mapping/core\_types/ > > > > So, my reading of this is that it should work - Andrei is, after all, > > searching on "fields": ["tags"] > > > > This may be a bug. > > > > Andrei, what happens if you search for: > > > > - "query\_string": "tags:coney\ island" > > - "query\_string": "tags:coney\ island" > > - "query\_string": "tags:(coney island)" > > > > I'm suggesting a few possibilities, because I'm not sure how to > > represent an embedded space in the query string. > > > > clint ---
**Author:** ![Andrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrei/32/2856_2.png) [@Andrei](https://discuss.elastic.co/u/Andrei)\ **Post date:** [January 16, 2011, 7:18pm UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/8 "2011-01-16T19:18:07Z")
My current query (before this change), actually does this: {"query\_string": { "fields": ["title", "notes", "tags"], "query": "coney island", "default\_operator": "AND", "use\_dis\_max": true }} All the fields had the "standard" analyzer. I wanted to change it so that the tags are matched completely, without breaking them up into words, while maintaining the current behavior with regard to title and notes. What is the best way of achieving this? -Andrei On Jan 16, 1:55 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > The query parser breaks down on whitespaces as well. So, the things that gets passed to the analyzer and then construct the query is "coney" and then "island" without doing the actual escaping. Not ideal, but thats how it works... ---
**Author:** ![Andrei](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrei/32/2856_2.png) [@Andrei](https://discuss.elastic.co/u/Andrei)\ **Post date:** [January 28, 2011, 8:35pm UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/9 "2011-01-28T20:35:37Z")
So, do I need to convert this into a dis\_max query? Because simply escaping the whitespace in the query string will not work for title or notes, because then it forces the words to be a phrase, basically. On Jan 16, 11:18 am, Andrei [and...@zmievski.org](mailto:and...@zmievski.org) wrote: > My current query (before this change), actually does this: > > {"query\_string": { > "fields": ["title", "notes", "tags"], > "query": "coney island", > "default\_operator": "AND", > "use\_dis\_max": true > > }} > > All the fields had the "standard" analyzer. I wanted to change it so > that the tags are matched completely, without breaking them up into > words, while maintaining the current behavior with regard to title and > notes. What is the best way of achieving this? > > -Andrei > > On Jan 16, 1:55 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > The query parser breaks down on whitespaces as well. So, the things that gets passed to the analyzer and then construct the query is "coney" and then "island" without doing the actual escaping. Not ideal, but thats how it works... ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:13am UTC](https://discuss.elastic.co/t/query-string-not-working-with-keyword-tokenizer/3778/10 "2017-07-06T04:13:12Z")