# Analyzers comparison **URL:** \ **Category:** Elasticsearch\ **Created:** [February 1, 2012, 10:02pm UTC](https://discuss.elastic.co/t/analyzers-comparison/6568 "2012-02-01T22:02:18Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![chimingc](https://avatars.discourse-cdn.com/v4/letter/c/ba9def/32.png) [@chimingc](https://discuss.elastic.co/u/chimingc)\ **Post date:** [February 1, 2012, 10:02pm UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/1 "2012-02-01T22:02:18Z")
Hi, I am trying to compare different analyzers to see which ones we should use. But my simple comparison shows me no difference between standard and keyword analyzers. I wonder what I am missing. This is the simple test I have. I have two fields using different analyzer (keyword vs standard): curl -XPOST localhost:9200/test -d '{ "mappings" : { "accounts" : { "properties" : { "keyword\_field" : { type: "string", analyzer : "keyword" }, "standard\_field" : { type: "string", analyzer : "standard" } } } } }' Then I have two records: curl -XPUT '[http://localhost:9200/test/accounts/1](http://localhost:9200/test/accounts/1)' -d '{ "keyword\_field" : "john doe" }' curl -XPUT '[http://localhost:9200/test/accounts/2](http://localhost:9200/test/accounts/2)' -d '{ "standard\_field" : "john doe" }' Finally, I use this query to get my results: curl "localhost:9200/test/accounts/\_search?pretty=true" -d '{ "query" : { "query\_string" : { "analyzer" : "keyword", //also tried "standard" "query" : "john doe" // also tried "john" or "doe" } } }' In the query\_string above, I have tried both keyword and standard. However, they always return me both documents with same score no matter which analyzer I have in my query\_string. I also tried replacing the query "john doe" with either only "john" or "doe", and it still returns me both documents regardless the analyzer in my query\_string. My understanding is that keyword analyzer does not tokenize while standard analyzer does. So I'd expect different search results. If this is not a correct test to see the difference between the two, what test can I use to see some differences so I know which one to use in my app? Thanks. ---
**Author:** ![Dan\_Everton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dan_everton/32/1865_2.png) [@Dan\_Everton](https://discuss.elastic.co/u/Dan_Everton)\ **Post date:** [February 1, 2012, 10:28pm UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/2 "2012-02-01T22:28:30Z")
Elasticsearch has an 'analyzer' API that you can use to see what the various analyzers do: see [http://www.elasticsearch.org/guide/reference/api/admin-indices-analyze.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-analyze.html). For your example: curl -XGET 'localhost:9200/test/\_analyze?pretty=true&text=john +doe&analyzer=standard' { "tokens" : [ { "token" : "john", "start\_offset" : 0, "end\_offset" : 4, "type" : "", "position" : 1 }, { "token" : "doe", "start\_offset" : 5, "end\_offset" : 8, "type" : "", "position" : 2 } ] } curl -XGET 'localhost:9200/test/\_analyze?pretty=true&text=john +doe&analyzer=keyword' { "tokens" : [ { "token" : "john doe", "start\_offset" : 0, "end\_offset" : 8, "type" : "word", "position" : 1 } ] } ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 1, 2012, 10:36pm UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/3 "2012-02-01T22:36:15Z")
Hi, See > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. [http://www.elasticsearch.org/guide/reference/api/admin-indices-analyze.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-analyze.html) You can test how the analyzer will tokenize your datas. It will help to understand what's going on. HTH David. Le 1 février 2012 à 23:02, chimingc [jchen@sugarcrm.com](mailto:jchen@sugarcrm.com) a écrit : > Hi, > > I am trying to compare different analyzers to see which ones we should > use. > But my simple comparison shows me no difference between standard and > keyword > analyzers. > I wonder what I am missing. > > This is the simple test I have. > > I have two fields using different analyzer (keyword vs standard): > curl -XPOST localhost:9200/test -d '{ > "mappings" : { > "accounts" : { > "properties" : { > "keyword\_field" : { type: "string", analyzer : "keyword" > }, > "standard\_field" : { type: "string", analyzer : > "standard" } > } > } > } > }' > > Then I have two records: > curl -XPUT '[http://localhost:9200/test/accounts/1](http://localhost:9200/test/accounts/1)' -d '{ > "keyword\_field" : "john doe" > }' > curl -XPUT '[http://localhost:9200/test/accounts/2](http://localhost:9200/test/accounts/2)' -d '{ > "standard\_field" : "john doe" > }' > > Finally, I use this query to get my results: > curl "localhost:9200/test/accounts/\_search?pretty=true" -d '{ > "query" : { > "query\_string" : { > "analyzer" : "keyword", //also tried "standard" > "query" : "john doe" // also tried "john" or "doe" > } > } > }' > > In the query\_string above, I have tried both keyword and standard. > However, they always return me both documents with same score no matter > which analyzer I have in my query\_string. > I also tried replacing the query "john doe" with either only "john" or > "doe", and it still returns me both documents regardless the analyzer in > my > query\_string. > > My understanding is that keyword analyzer does not tokenize while > standard > analyzer does. So I'd expect different search results. > > If this is not a correct test to see the difference between the two, what > test can I use to see some differences so I know which one to use in my > app? > > Thanks. > > -- > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/Analyzers-comparison-tp3708214p3708214.html](http://elasticsearch-users.115913.n3.nabble.com/Analyzers-comparison-tp3708214p3708214.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). -- David Pilato [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : @dadoonet ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 1, 2012, 10:37pm UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/4 "2012-02-01T22:37:22Z")
Too late ! Dan was the first ! 😉 Le 1 février 2012 à 23:36, "[david@pilato.fr](mailto:david@pilato.fr)" [david@pilato.fr](mailto:david@pilato.fr) a écrit : > Hi, > > See > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-analyze.html) > [http://www.elasticsearch.org/guide/reference/api/admin-indices-analyze.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-analyze.html) > You can test how the analyzer will tokenize your datas. > > It will help to understand what's going on. > > HTH > David. > > Le 1 février 2012 à 23:02, chimingc [jchen@sugarcrm.com](mailto:jchen@sugarcrm.com) a écrit : > > > Hi, > > > > I am trying to compare different analyzers to see which ones we should > > use. > > But my simple comparison shows me no difference between standard and > > keyword > > analyzers. > > I wonder what I am missing. > > > > This is the simple test I have. > > > > I have two fields using different analyzer (keyword vs standard): > > curl -XPOST localhost:9200/test -d '{ > > "mappings" : { > > "accounts" : { > > "properties" : { > > "keyword\_field" : { type: "string", analyzer : "keyword" > > }, > > "standard\_field" : { type: "string", analyzer : > > "standard" } > > } > > } > > } > > }' > > > > Then I have two records: > > curl -XPUT '[http://localhost:9200/test/accounts/1](http://localhost:9200/test/accounts/1)' -d '{ > > "keyword\_field" : "john doe" > > }' > > curl -XPUT '[http://localhost:9200/test/accounts/2](http://localhost:9200/test/accounts/2)' -d '{ > > "standard\_field" : "john doe" > > }' > > > > Finally, I use this query to get my results: > > curl "localhost:9200/test/accounts/\_search?pretty=true" -d '{ > > "query" : { > > "query\_string" : { > > "analyzer" : "keyword", //also tried "standard" > > "query" : "john doe" // also tried "john" or "doe" > > } > > } > > }' > > > > In the query\_string above, I have tried both keyword and standard. > > However, they always return me both documents with same score no matter > > which analyzer I have in my query\_string. > > I also tried replacing the query "john doe" with either only "john" or > > "doe", and it still returns me both documents regardless the analyzer in > > my > > query\_string. > > > > My understanding is that keyword analyzer does not tokenize while > > standard > > analyzer does. So I'd expect different search results. > > > > If this is not a correct test to see the difference between the two, what > > test can I use to see some differences so I know which one to use in my > > app? > > > > Thanks. > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/Analyzers-comparison-tp3708214p3708214.html](http://elasticsearch-users.115913.n3.nabble.com/Analyzers-comparison-tp3708214p3708214.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). > > -- > David Pilato > [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) > Twitter : @dadoonet -- David Pilato [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : @dadoonet ---
**Author:** ![chimingc](https://avatars.discourse-cdn.com/v4/letter/c/ba9def/32.png) [@chimingc](https://discuss.elastic.co/u/chimingc)\ **Post date:** [February 1, 2012, 10:47pm UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/5 "2012-02-01T22:47:58Z")
Thanks guys. I am aware of the analyzers API and I do see that they tokenize strings differently. What puzzles me is that the search results. I couldn't differentiate the two from the search results. Perhaps I didn't have the correct query\_string but I couldn't find any other way to pass an analyzer in the query\_string. ---
**Author:** ![chimingc](https://avatars.discourse-cdn.com/v4/letter/c/ba9def/32.png) [@chimingc](https://discuss.elastic.co/u/chimingc)\ **Post date:** [February 2, 2012, 12:22am UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/6 "2012-02-02T00:22:47Z")
I just tried some query that is not supposed to work: curl "localhost:9200/test/accounts/\_search?pretty=true" -d '{ "query" : { "query\_string" : { "analyzer" : "xyz", "query" : "john and his friend jane" } } }' And that still returns me both records. As you can see "xyz" is not even a valid analyzer. Is it possible that the analyzer in query\_string is totally ignored? From [http://www.elasticsearch.org/guide/reference/query-dsl/query-string-query.html](http://www.elasticsearch.org/guide/reference/query-dsl/query-string-query.html) it seems analyzer is a valid parameter, though. ---
**Author:** ![inflynx](https://avatars.discourse-cdn.com/v4/letter/i/ecae2f/32.png) [@inflynx](https://discuss.elastic.co/u/inflynx)\ **Post date:** [February 2, 2012, 12:29am UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/7 "2012-02-02T00:29:51Z")
On 01/02/2012 22:02, chimingc wrote: > Hi, > > I am trying to compare different analyzers to see which ones we should use. > But my simple comparison shows me no difference between standard and keyword > analyzers. > I wonder what I am missing. > > This is the simple test I have. > > I have two fields using different analyzer (keyword vs standard): > curl -XPOST localhost:9200/test -d '{ > "mappings" : { > "accounts" : { > "properties" : { > "keyword\_field" : { type: "string", analyzer : "keyword" }, > "standard\_field" : { type: "string", analyzer : "standard" } > } > } > } > }' > > Then I have two records: > curl -XPUT '[http://localhost:9200/test/accounts/1](http://localhost:9200/test/accounts/1)' -d '{ > "keyword\_field" : "john doe" > }' > curl -XPUT '[http://localhost:9200/test/accounts/2](http://localhost:9200/test/accounts/2)' -d '{ > "standard\_field" : "john doe" > }' > > Finally, I use this query to get my results: > curl "localhost:9200/test/accounts/\_search?pretty=true" -d '{ > "query" : { > "query\_string" : { > "analyzer" : "keyword", //also tried "standard" > "query" : "john doe" // also tried "john" or "doe" > } > } > }' > > In the query\_string above, I have tried both keyword and standard. > However, they always return me both documents with same score no matter > which analyzer I have in my query\_string. > I also tried replacing the query "john doe" with either only "john" or > "doe", and it still returns me both documents regardless the analyzer in my > query\_string. > > My understanding is that keyword analyzer does not tokenize while standard > analyzer does. So I'd expect different search results. > > If this is not a correct test to see the difference between the two, what > test can I use to see some differences so I know which one to use in my app? > > Thanks. > > -- > View this message in context: [http://elasticsearch-users.115913.n3.nabble.com/Analyzers-comparison-tp3708214p3708214.html](http://elasticsearch-users.115913.n3.nabble.com/Analyzers-comparison-tp3708214p3708214.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![chimingc](https://avatars.discourse-cdn.com/v4/letter/c/ba9def/32.png) [@chimingc](https://discuss.elastic.co/u/chimingc)\ **Post date:** [February 2, 2012, 1:35am UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/8 "2012-02-02T01:35:32Z")
Please note that I am using elasticsearch-0.18.6, which should already include this bug fix: > ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 5, 2012, 11:16am UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/9 "2012-02-05T11:16:23Z")
Hi, Here is what happens. First, you use a different analyzer on different fields in the json, but then you use a query\_string which by default searches on the \_all field (which has its own analyzer setting). The query\_string syntax breaks the text provided by whitespace, _regardless_ of the analyzer provided, and then applies the analyzer to the result of it. This is done in order to parse the different operators and query language it has. You can wrap the text you want to search on in `\"` and then it will be treated as a phrase which will then by analyzed completely using the provided analyzer. ``` The text query can be simpler in those cases, as it treats the whole text and then analyzes it to generate (different) type of queries. ``` -shay.banon On Thursday, February 2, 2012 at 12:02 AM, chimingc wrote: > curl "localhost:9200/test/accounts/\_search?pretty=true" -d '{ > "query" : { > "query\_string" : { > "analyzer" : "keyword", //also tried "standard" > "query" : "john doe" // also tried "john" or "doe" > } > } > }' ---
**Author:** ![chimingc](https://avatars.discourse-cdn.com/v4/letter/c/ba9def/32.png) [@chimingc](https://discuss.elastic.co/u/chimingc)\ **Post date:** [February 5, 2012, 11:08pm UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/10 "2012-02-05T23:08:40Z")
So is there any way to search on "all fields" instead of "\_all field"? Or do I have to explicitly list all my fields (can be many) in the query\_string? My goal is use different analyzer on different field, and when I search, I want to search on all fields with their own analyzer. Thanks. From: "kimchy [via ElasticSearch Users]" \<[ml-node+s115913n3717037h51@n3.nabble.com](mailto:ml-node+s115913n3717037h51@n3.nabble.com)[mailto:ml-node+s115913n3717037h51@n3.nabble.com](mailto:ml-node+s115913n3717037h51@n3.nabble.com)\> Date: Sun, 5 Feb 2012 05:16:59 -0600 To: Jimmy Chen \<[jchen@sugarcrm.com](mailto:jchen@sugarcrm.com)[mailto:jchen@sugarcrm.com](mailto:jchen@sugarcrm.com)\> Subject: Re: Analyzers comparison Here is what happens. First, you use a different analyzer on different fields in the json, but then you use a query\_string which by default searches on the \_all field (which has its own analyzer setting). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [February 7, 2012, 10:01am UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/11 "2012-02-07T10:01:50Z")
Yes, you will need to specify several fields if you want to search across specific ones. The more though, the slower the search will be. On Monday, February 6, 2012 at 1:08 AM, chimingc wrote: > So is there any way to search on "all fields" instead of "\_all field"? > Or do I have to explicitly list all my fields (can be many) in the query\_string? > > My goal is use different analyzer on different field, and when I search, I want to search on all fields with their own analyzer. > Thanks. > > From: "kimchy [via Elasticsearch Users]" \<[hidden email] (/user/SendEmail.jtp?type=node&node=3718456&i=0)\> > Date: Sun, 5 Feb 2012 05:16:59 -0600 > To: Jimmy Chen \<[hidden email] (/user/SendEmail.jtp?type=node&node=3718456&i=1)\> > Subject: Re: Analyzers comparison > > Here is what happens. First, you use a different analyzer on different fields in the json, but then you use a query\_string which by default searches on the \_all field (which has its own analyzer setting). > > View this message in context: Re: Analyzers comparison ([http://elasticsearch-users.115913.n3.nabble.com/Analyzers-comparison-tp3708214p3718456.html](http://elasticsearch-users.115913.n3.nabble.com/Analyzers-comparison-tp3708214p3718456.html)) > Sent from the Elasticsearch Users mailing list archive ([http://elasticsearch-users.115913.n3.nabble.com/](http://elasticsearch-users.115913.n3.nabble.com/)) at [Nabble.com](http://Nabble.com) ([http://Nabble.com](http://Nabble.com)). ---
**Author:** ![Pravin\_2](https://avatars.discourse-cdn.com/v4/letter/p/e8c25b/32.png) [@Pravin\_2](https://discuss.elastic.co/u/Pravin_2)\ **Post date:** [December 6, 2012, 1:05pm UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/12 "2012-12-06T13:05:44Z")
Hi Kimchy, I was facing the same problem, But now I am able to search as per your below solution, But one more problem I am facing is that if I use "_" with my query which contains hyphen ("-"). It breaks the second word after hyphen and gives the result started with that second word. It should consider the complete word and gives the result with wildcard character "_". Here is my query, { "query": { "query\_string": { "default\_field": "\_all", "default\_operator": "AND", "analyze\_wildcard": true, "query": "QNAM-45\*" } } } As I explained this query returns me the result started with 45. But I need the results which will start with the complete word ("QNAM-45) Please give me the solution for this. Your help will be highly appreciable. Thanks & Regards, Pravin On Sunday, February 5, 2012 1:16:23 PM UTC+2, kimchy wrote: > Hi, > > Here is what happens. First, you use a different analyzer on different > fields in the json, but then you use a query\_string which by default > searches on the \_all field (which has its own analyzer setting). > > The query\_string syntax breaks the text provided by whitespace, > _regardless_ of the analyzer provided, and then applies the analyzer to the > result of it. This is done in order to parse the different operators and > query language it has. You can wrap the text you want to search on in `\"` > and then it will be treated as a phrase which will then by analyzed > completely using the provided analyzer. > > ``` > The text query can be simpler in those cases, as it treats the whole > > ``` > > text and then analyzes it to generate (different) type of queries. > > -shay.banon > > On Thursday, February 2, 2012 at 12:02 AM, chimingc wrote: > > curl "localhost:9200/test/accounts/\_search?pretty=true" -d '{ > "query" : { > "query\_string" : { > "analyzer" : "keyword", //also tried "standard" > "query" : "john doe" // also tried "john" or "doe" > } > } > }' -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:01am UTC](https://discuss.elastic.co/t/analyzers-comparison/6568/13 "2017-07-06T03:01:05Z")