# Querystring search: Tokens are out of order

**URL:** <https://discuss.elastic.co/t/querystring-search-tokens-are-out-of-order/23238>\
**Category:** Elasticsearch\
**Created:** [April 14, 2015, 7:03pm UTC](https://discuss.elastic.co/t/querystring-search-tokens-are-out-of-order/23238 "2015-04-14T19:03:01Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Dave\_Reed](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@Dave\_Reed](https://discuss.elastic.co/u/Dave_Reed)\
**Post date:** [April 14, 2015, 7:03pm UTC](https://discuss.elastic.co/t/querystring-search-tokens-are-out-of-order/23238/1 "2015-04-14T19:03:01Z")

</div>

I have the following search:

{  
"query": {  
"filtered": {  
"query": {  
"query\_string": {  
"default\_operator": "AND",  
"query": "details:foo\-bar"  
}  
},  
"filter": {  
"term": {  
"deleted": false  
}  
}  
}  
}  
}

The details field is analyzed using pattern tokenizer, as so:

settings: {  
index.analysis.analyzer.letterordigit.pattern: "[^\p{L}\p{N}]+",  
index.analysis.analyzer.letterordigit.type: "pattern"  
}

This breaks the field into tokens separated by any non-letter or  
non-numeric character.

But the user is searching for "foo-bar" which contains a non alphanumeric  
character. I assume, but correct me if I'm wrong, that ES will apply the  
same analyzer to that string. So it is broken into two tokens: ["foo",  
"bar"], and then the default\_operator kicks in and essentially turns the  
query into "details:foo AND detail:bar".

My problem is that it will match documents containing "foo xyz bar" and  
"bar xyz foo" -- in the latter case, the tokens are in the reverse order  
from the user's search. I'm fine with it matching the former, but it's a  
stretch to convince the user that the latter is intended.

The search string is provided by the user, so I can't really build a  
complex query with different query types, hence the basic querystring  
search.

Any advice or corrections to my assumptions is appreciated!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![James\_Macdonald](https://avatars.discourse-cdn.com/v4/letter/j/f19dbf/32.png) [@James\_Macdonald](https://discuss.elastic.co/u/James_Macdonald)\
**Post date:** [April 14, 2015, 8:14pm UTC](https://discuss.elastic.co/t/querystring-search-tokens-are-out-of-order/23238/2 "2015-04-14T20:14:36Z")

</div>

You analysis of what is going on sounds correct. However, Elasticsearch's  
results are also correct. When it analyzes the search string, your query  
becomes a match query on "foo" AND "bar", which matches any document  
containing both of those terms. Most queries against analyzed fields do not  
respect the original ordering of the terms.

One thing you could try is looking into the match\_phrase query (  
[Phrase Matching | Elasticsearch: The Definitive Guide [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/guide/master/phrase-matching.html))  
which is aware of the ordering of the terms. Using the base match\_phrase  
query for "foo bar" will not match either "foo xyz bar" or "bar xyz foo".  
If you still need to match things like "foo xyz bar" you may be able to do  
that using the slop parameter, depending on what exactly the use case is.

James

On Tue, Apr 14, 2015 at 2:03 PM, Dave Reed [infinity88@gmail.com](mailto:infinity88@gmail.com) wrote:

> I have the following search:
> 
> {  
> "query": {  
> "filtered": {  
> "query": {  
> "query\_string": {  
> "default\_operator": "AND",  
> "query": "details:foo\-bar"  
> }  
> },  
> "filter": {  
> "term": {  
> "deleted": false  
> }  
> }  
> }  
> }  
> }
> 
> The details field is analyzed using pattern tokenizer, as so:
> 
> settings: {  
> index.analysis.analyzer.letterordigit.pattern: "[^\p{L}\p{N}]+",  
> index.analysis.analyzer.letterordigit.type: "pattern"  
> }
> 
> This breaks the field into tokens separated by any non-letter or  
> non-numeric character.
> 
> But the user is searching for "foo-bar" which contains a non alphanumeric  
> character. I assume, but correct me if I'm wrong, that ES will apply the  
> same analyzer to that string. So it is broken into two tokens: ["foo",  
> "bar"], and then the default\_operator kicks in and essentially turns the  
> query into "details:foo AND detail:bar".
> 
> My problem is that it will match documents containing "foo xyz bar" and  
> "bar xyz foo" -- in the latter case, the tokens are in the reverse order  
> from the user's search. I'm fine with it matching the former, but it's a  
> stretch to convince the user that the latter is intended.
> 
> The search string is provided by the user, so I can't really build a  
> complex query with different query types, hence the basic querystring  
> search.
> 
> Any advice or corrections to my assumptions is appreciated!
> 
> --  
> You received this message because you are subscribed to the Google Groups  
> "elasticsearch" group.  
> To unsubscribe from this group and stop receiving emails from it, send an  
> email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> To view this discussion on the web visit  
> [https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com)  
> [https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com?utm_medium=email&utm_source=footer)  
> .  
> For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAABsnTZWNp65WzwYsZVZz%3DiHon7WW90EO8SUKbnB4aHuKcd-og%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAABsnTZWNp65WzwYsZVZz%3DiHon7WW90EO8SUKbnB4aHuKcd-og%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Dave\_Reed](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@Dave\_Reed](https://discuss.elastic.co/u/Dave_Reed)\
**Post date:** [April 14, 2015, 8:34pm UTC](https://discuss.elastic.co/t/querystring-search-tokens-are-out-of-order/23238/3 "2015-04-14T20:34:28Z")

</div>

Thanks, though unless I am misunderstanding it, the docs imply otherwise:

For example, from:

> **[Query string query | Elasticsearch Guide \[8.11\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html)**

The query string is parsed into a series of _terms_ and _operators_. A term

> can be a single word — quick or brown — or a phrase, surrounded by double  
> quotes — "quick brown" — which searches for all the words in the phrase,  
> in the same order.

So what gives? 🙂

On Tuesday, April 14, 2015 at 1:15:24 PM UTC-7, James Macdonald wrote:

> You analysis of what is going on sounds correct. However, Elasticsearch's  
> results are also correct. When it analyzes the search string, your query  
> becomes a match query on "foo" AND "bar", which matches any document  
> containing both of those terms. Most queries against analyzed fields do not  
> respect the original ordering of the terms.
> 
> One thing you could try is looking into the match\_phrase query (  
> [Phrase Matching | Elasticsearch: The Definitive Guide [master] | Elastic](http://www.elastic.co/guide/en/elasticsearch/guide/master/phrase-matching.html))  
> which is aware of the ordering of the terms. Using the base match\_phrase  
> query for "foo bar" will not match either "foo xyz bar" or "bar xyz foo".  
> If you still need to match things like "foo xyz bar" you may be able to do  
> that using the slop parameter, depending on what exactly the use case is.
> 
> James
> 
> On Tue, Apr 14, 2015 at 2:03 PM, Dave Reed \<[infin...@gmail.com](mailto:infin...@gmail.com)  
> \<javascript:\>\> wrote:
> 
> > I have the following search:
> > 
> > {  
> > "query": {  
> > "filtered": {  
> > "query": {  
> > "query\_string": {  
> > "default\_operator": "AND",  
> > "query": "details:foo\-bar"  
> > }  
> > },  
> > "filter": {  
> > "term": {  
> > "deleted": false  
> > }  
> > }  
> > }  
> > }  
> > }
> > 
> > The details field is analyzed using pattern tokenizer, as so:
> > 
> > settings: {  
> > index.analysis.analyzer.letterordigit.pattern: "[^\p{L}\p{N}]+",  
> > index.analysis.analyzer.letterordigit.type: "pattern"  
> > }
> > 
> > This breaks the field into tokens separated by any non-letter or  
> > non-numeric character.
> > 
> > But the user is searching for "foo-bar" which contains a non alphanumeric  
> > character. I assume, but correct me if I'm wrong, that ES will apply the  
> > same analyzer to that string. So it is broken into two tokens: ["foo",  
> > "bar"], and then the default\_operator kicks in and essentially turns the  
> > query into "details:foo AND detail:bar".
> > 
> > My problem is that it will match documents containing "foo xyz bar" and  
> > "bar xyz foo" -- in the latter case, the tokens are in the reverse order  
> > from the user's search. I'm fine with it matching the former, but it's a  
> > stretch to convince the user that the latter is intended.
> > 
> > The search string is provided by the user, so I can't really build a  
> > complex query with different query types, hence the basic querystring  
> > search.
> > 
> > Any advice or corrections to my assumptions is appreciated!
> > 
> > --  
> > You received this message because you are subscribed to the Google Groups  
> > "elasticsearch" group.  
> > To unsubscribe from this group and stop receiving emails from it, send an  
> > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \<javascript:\>.  
> > To view this discussion on the web visit  
> > [https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com)  
> > [https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4a204214-f209-48dd-a13a-96463609ad7d%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > .  
> > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7a355b94-358f-4c5a-ac16-31ac7a0c0abe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/7a355b94-358f-4c5a-ac16-31ac7a0c0abe%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Dave\_Reed](https://avatars.discourse-cdn.com/v4/letter/d/958977/32.png) [@Dave\_Reed](https://discuss.elastic.co/u/Dave_Reed)\
**Post date:** [April 14, 2015, 8:38pm UTC](https://discuss.elastic.co/t/querystring-search-tokens-are-out-of-order/23238/4 "2015-04-14T20:38:01Z")

</div>

To perhaps answer my own question, I think I understand the difference.

details:"foo bar"

Would search for the tokens in the same order (implied by the docs I  
referenced). But

details:foo-bar

Would not honor the order. The quotes have more meaning than to enclose the  
phrase... if that is true then these two queries are not the same, which is  
different than I thought:

details:foo\ bar  
!=  
details:"foo bar"

Or am I barking up the wrong tree...

On Tuesday, April 14, 2015 at 1:34:28 PM UTC-7, Dave Reed wrote:

> Thanks, though unless I am misunderstanding it, the docs imply otherwise:
> 
> For example, from:
> 
> [Query string query | Elasticsearch Guide [8.11] | Elastic](http://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html)
> 
> The query string is parsed into a series of _terms_ and _operators_. A
> 
> > term can be a single word — quick or brown — or a phrase, surrounded by  
> > double quotes — "quick brown" — which searches for all the words in the  
> > phrase, in the same order.
> 
> So what gives? 🙂
> 
> >

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b28591e3-3818-4b12-8a22-cac466c9ec7c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b28591e3-3818-4b12-8a22-cac466c9ec7c%40googlegroups.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [April 15, 2015, 8:32am UTC](https://discuss.elastic.co/t/querystring-search-tokens-are-out-of-order/23238/5 "2015-04-15T08:32:37Z")

</div>

You understanding is correct. The former will be translated into a Lucene  
phrase query, which uses the term doc positions to find matches.

Both query terms are analyzed, but the latter will simply be a bag-of-words  
query, which ignores positions.

Cheers,

Ivan  
On Apr 14, 2015 10:38 PM, "Dave Reed" [infinity88@gmail.com](mailto:infinity88@gmail.com) wrote:

> To perhaps answer my own question, I think I understand the difference.
> 
> details:"foo bar"
> 
> Would search for the tokens in the same order (implied by the docs I  
> referenced). But
> 
> details:foo-bar
> 
> Would not honor the order. The quotes have more meaning than to enclose  
> the phrase... if that is true then these two queries are not the same,  
> which is different than I thought:
> 
> details:foo\ bar  
> !=  
> details:"foo bar"
> 
> Or am I barking up the wrong tree...
> 
> On Tuesday, April 14, 2015 at 1:34:28 PM UTC-7, Dave Reed wrote:
> 
> > Thanks, though unless I am misunderstanding it, the docs imply otherwise:
> > 
> > For example, from:  
> > [Elasticsearch Guide | Elastic](http://www.elastic.co/guide/en/elasticsearch/reference/)  
> > current/query-dsl-query-string-query.html
> > 
> > The query string is parsed into a series of _terms_ and _operators_. A
> > 
> > > term can be a single word — quick or brown — or a phrase, surrounded by  
> > > double quotes — "quick brown" — which searches for all the words in the  
> > > phrase, in the same order.
> > 
> > So what gives? 🙂
> > 
> > > --  
> > > You received this message because you are subscribed to the Google Groups  
> > > "elasticsearch" group.  
> > > To unsubscribe from this group and stop receiving emails from it, send an  
> > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
> > > To view this discussion on the web visit  
> > > [https://groups.google.com/d/msgid/elasticsearch/b28591e3-3818-4b12-8a22-cac466c9ec7c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b28591e3-3818-4b12-8a22-cac466c9ec7c%40googlegroups.com)  
> > > [https://groups.google.com/d/msgid/elasticsearch/b28591e3-3818-4b12-8a22-cac466c9ec7c%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/b28591e3-3818-4b12-8a22-cac466c9ec7c%40googlegroups.com?utm_medium=email&utm_source=footer)  
> > > .  
> > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBZOjqZ6xU8Y2%3Dh6BmBWOqms53yrix5eJsWXq9E6meYbg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQBZOjqZ6xU8Y2%3Dh6BmBWOqms53yrix5eJsWXq9E6meYbg%40mail.gmail.com).  
For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 12:19am UTC](https://discuss.elastic.co/t/querystring-search-tokens-are-out-of-order/23238/6 "2017-07-06T00:19:39Z")

</div>