BoolFilterBuilder does not merge multiple must-Clauses into array, but BoolQueryBuilder does

Elastic Stack Elasticsearch
1

Hey everybody,

during my experiments with elasticsearch I've found differing behaviour
using the Java-API-Builders for BoolFilters and BoolQueries.

When I try to add multiple must clauses to both of them, the BooleanQuery
seemingly works correct in merging multiple must-terms into an array, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(boolQuery().must(termQuery("f1",

"v1")).must(termQuery("f2", "v2")));

s.toString returns:
{
"query" : {
"bool" : {
"must" : [ {
"term" : {
"f1" : "v1"
}
}, {
"term" : {
"f2" : "v2"
}
} ]
}
}
}

But, when I try to do the equivalent for a BooleanFilter, multiple
must-terms collide and only one of them is preserved in the
Json-representation, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(constantScoreQuery(boolFilter().must(termFilter("f1",

"v1")).must(termFilter("f2", "v2"))));

s.toString returns:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : {
"term" : {
"f1" : "v1"
}
},
"must" : {
"term" : {
"f2" : "v2"
}
}
}
}
}
}
}

Here both must-Terms within the Hash collide.

Checking the source, I found that the implementation of both classes differ
substantially, so that I suspect there is a misunderstanding on my side.

Any help is highly appreciated!

Thank you very much.
Björn Hachmann

2

Both bool query and filer support providing multiple "must" filter each
under its own "must" element, or a single "must" element with an array of
filters. The bool query simply generates the array option, and the bool
filter generates "must" for each clause, but both are supported and produce
the same behavior.

On Mon, Mar 19, 2012 at 2:21 PM, Hachmaninow b.hachmann@googlemail.comwrote:

Hey everybody,

during my experiments with elasticsearch I've found differing behaviour
using the Java-API-Builders for BoolFilters and BoolQueries.

When I try to add multiple must clauses to both of them, the BooleanQuery
seemingly works correct in merging multiple must-terms into an array, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(boolQuery().must(termQuery("f1",

"v1")).must(termQuery("f2", "v2")));

s.toString returns:
{
"query" : {
"bool" : {
"must" : [ {
"term" : {
"f1" : "v1"
}
}, {
"term" : {
"f2" : "v2"
}
} ]
}
}
}

But, when I try to do the equivalent for a BooleanFilter, multiple
must-terms collide and only one of them is preserved in the
Json-representation, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(constantScoreQuery(boolFilter().must(termFilter("f1",

"v1")).must(termFilter("f2", "v2"))));

s.toString returns:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : {
"term" : {
"f1" : "v1"
}
},
"must" : {
"term" : {
"f2" : "v2"
}
}
}
}
}
}
}

Here both must-Terms within the Hash collide.

Checking the source, I found that the implementation of both classes
differ substantially, so that I suspect there is a misunderstanding on my
side.

Any help is highly appreciated!

Thank you very much.
Björn Hachmann

3

Hey Kimchy,

sorry for coming back to this late. Thank you very much for this answer!

Do you mean the SearchRequestBuilder is fine and will return the correct
answers when used to execute a search... that is most important, in fact.
But this in mind, we should not rely on the toString method of the builder
to return an equivalent json representation of the search. Is there any
hidden reason for this limitation? I mean - this would be quite useful...

To clearify: When I use tool like jsonlint.com to validate the above json,
I got as expected:
{
"query": {
"constant_score": {
"filter": {
"bool": {
"must": {
"term": {
"f2": "v2"
}
}
}
}
}
}
}

So one of the must-terms got lost.

Thank you.

Cheers
Björn Hachmann

Am Dienstag, 20. März 2012 11:02:09 UTC+1 schrieb kimchy:

Both bool query and filer support providing multiple "must" filter each
under its own "must" element, or a single "must" element with an array of
filters. The bool query simply generates the array option, and the bool
filter generates "must" for each clause, but both are supported and produce
the same behavior.

On Mon, Mar 19, 2012 at 2:21 PM, Hachmaninow b.hachmann@googlemail.comwrote:

Hey everybody,

during my experiments with elasticsearch I've found differing behaviour
using the Java-API-Builders for BoolFilters and BoolQueries.

When I try to add multiple must clauses to both of them, the BooleanQuery
seemingly works correct in merging multiple must-terms into an array, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(boolQuery().must(termQuery("f1",

"v1")).must(termQuery("f2", "v2")));

s.toString returns:
{
"query" : {
"bool" : {
"must" : [ {
"term" : {
"f1" : "v1"
}
}, {
"term" : {
"f2" : "v2"
}
} ]
}
}
}

But, when I try to do the equivalent for a BooleanFilter, multiple
must-terms collide and only one of them is preserved in the
Json-representation, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(constantScoreQuery(boolFilter().must(termFilter("f1",

"v1")).must(termFilter("f2", "v2"))));

s.toString returns:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : {
"term" : {
"f1" : "v1"
}
},
"must" : {
"term" : {
"f2" : "v2"
}
}
}
}
}
}
}

Here both must-Terms within the Hash collide.

Checking the source, I found that the implementation of both classes
differ substantially, so that I suspect there is a misunderstanding on my
side.

Any help is highly appreciated!

Thank you very much.
Björn Hachmann

4

What I meant is that both formats works, the {"must" : [{}, {}, {}]}, and
{"must" : {}, "must" : {}}.

On Thu, Mar 22, 2012 at 5:32 PM, Hachmaninow b.hachmann@googlemail.comwrote:

Hey Kimchy,

sorry for coming back to this late. Thank you very much for this answer!

Do you mean the SearchRequestBuilder is fine and will return the correct
answers when used to execute a search... that is most important, in fact.
But this in mind, we should not rely on the toString method of the builder
to return an equivalent json representation of the search. Is there any
hidden reason for this limitation? I mean - this would be quite useful...

To clearify: When I use tool like jsonlint.com to validate the above
json, I got as expected:

{
"query": {
"constant_score": {
"filter": {
"bool": {
"must": {
"term": {
"f2": "v2"
}
}
}
}
}
}
}

So one of the must-terms got lost.

Thank you.

Cheers
Björn Hachmann

Am Dienstag, 20. März 2012 11:02:09 UTC+1 schrieb kimchy:

Both bool query and filer support providing multiple "must" filter each
under its own "must" element, or a single "must" element with an array of
filters. The bool query simply generates the array option, and the bool
filter generates "must" for each clause, but both are supported and produce
the same behavior.

On Mon, Mar 19, 2012 at 2:21 PM, Hachmaninow b.hachmann@googlemail.comwrote:

Hey everybody,

during my experiments with elasticsearch I've found differing behaviour
using the Java-API-Builders for BoolFilters and BoolQueries.

When I try to add multiple must clauses to both of them, the
BooleanQuery seemingly works correct in merging multiple must-terms into an
array, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(boolQuery().must(**termQuery("f1",

"v1")).must(termQuery("f2", "v2")));

s.toString returns:
{
"query" : {
"bool" : {
"must" : [ {
"term" : {
"f1" : "v1"
}
}, {
"term" : {
"f2" : "v2"
}
} ]
}
}
}

But, when I try to do the equivalent for a BooleanFilter, multiple
must-terms collide and only one of them is preserved in the
Json-representation, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(constantScoreQuery(**boolFilter().must(termFilter("**f1",

"v1")).must(termFilter("f2", "v2"))));

s.toString returns:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : {
"term" : {
"f1" : "v1"
}
},
"must" : {
"term" : {
"f2" : "v2"
}
}
}
}
}
}
}

Here both must-Terms within the Hash collide.

Checking the source, I found that the implementation of both classes
differ substantially, so that I suspect there is a misunderstanding on my
side.

Any help is highly appreciated!

Thank you very much.
Björn Hachmann

5

Ok. Thanks a lot for your time!

Am Sonntag, 25. März 2012 13:33:08 UTC+2 schrieb kimchy:

What I meant is that both formats works, the {"must" : [{}, {}, {}]}, and
{"must" : {}, "must" : {}}.

On Thu, Mar 22, 2012 at 5:32 PM, Hachmaninow b.hachmann@googlemail.comwrote:

Hey Kimchy,

sorry for coming back to this late. Thank you very much for this answer!

Do you mean the SearchRequestBuilder is fine and will return the correct
answers when used to execute a search... that is most important, in fact.
But this in mind, we should not rely on the toString method of the
builder to return an equivalent json representation of the search. Is there
any hidden reason for this limitation? I mean - this would be quite
useful...

To clearify: When I use tool like jsonlint.com to validate the above
json, I got as expected:

{
"query": {
"constant_score": {
"filter": {
"bool": {
"must": {
"term": {
"f2": "v2"
}
}
}
}
}
}
}

So one of the must-terms got lost.

Thank you.

Cheers
Björn Hachmann

Am Dienstag, 20. März 2012 11:02:09 UTC+1 schrieb kimchy:

Both bool query and filer support providing multiple "must" filter each
under its own "must" element, or a single "must" element with an array of
filters. The bool query simply generates the array option, and the bool
filter generates "must" for each clause, but both are supported and produce
the same behavior.

On Mon, Mar 19, 2012 at 2:21 PM, Hachmaninow b.hachmann@googlemail.comwrote:

Hey everybody,

during my experiments with elasticsearch I've found differing behaviour
using the Java-API-Builders for BoolFilters and BoolQueries.

When I try to add multiple must clauses to both of them, the
BooleanQuery seemingly works correct in merging multiple must-terms into an
array, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(boolQuery().must(**te​rmQuery("f1",

"v1")).must(termQuery("f2", "v2")));

s.toString returns:
{
"query" : {
"bool" : {
"must" : [ {
"term" : {
"f1" : "v1"
}
}, {
"term" : {
"f2" : "v2"
}
} ]
}
}
}

But, when I try to do the equivalent for a BooleanFilter, multiple
must-terms collide and only one of them is preserved in the
Json-representation, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(constantScoreQuery(**​boolFilter().must(termFilter("

**​f1", "v1")).must(termFilter("f2", "v2"))));

s.toString returns:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : {
"term" : {
"f1" : "v1"
}
},
"must" : {
"term" : {
"f2" : "v2"
}
}
}
}
}
}
}

Here both must-Terms within the Hash collide.

Checking the source, I found that the implementation of both classes
differ substantially, so that I suspect there is a misunderstanding on my
side.

Any help is highly appreciated!

Thank you very much.
Björn Hachmann

6

hmm, I was expecting these clauses to be the same:
{bool:{should:{1},should{2}} and {bool:{should:[{1},{2}]}}

but it doesn't look to be the case actually. Here is the full query (well,
filter to be precise) example:

{"filtered":{
"query":{
"bool":{
"must":{
"match_all":{}
}
}
},
"filter":{
"bool":{
"must":{
"bool":{
"should":{
"term":{
"f1":"v1"
}
},
"should":{
"term":{
"f2":"v2"
}
}
}
}
}
}
}}

result shows that whatever 'should' clause is the last one overrides all
what was before it.

but when i manually fix query via web console to look like:

{"filtered":{
"query":{
"bool":{
"must":{
"match_all":{}
}
}
},
"filter":{
"bool":{
"must":{
"bool":{
"should":[
{
"term":{
"f1":"v1"
}
},
{
"term":{
"f2":"v2"
}
}
]
}
}
}
}
}}

it correctly shows documents matching either f1:v1 OR f2:v2
It's really weird and unexpected behavior, and it seems like it's no way i
can produce array-like query format from Java API. reproduced in v0.19.8.

Best regards,
Ivan.

On Monday, March 19, 2012 2:21:23 PM UTC+2, Hachmaninow wrote:

Hey everybody,

during my experiments with elasticsearch I've found differing behaviour
using the Java-API-Builders for BoolFilters and BoolQueries.

When I try to add multiple must clauses to both of them, the BooleanQuery
seemingly works correct in merging multiple must-terms into an array, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(boolQuery().must(termQuery("f1",

"v1")).must(termQuery("f2", "v2")));

s.toString returns:
{
"query" : {
"bool" : {
"must" : [ {
"term" : {
"f1" : "v1"
}
}, {
"term" : {
"f2" : "v2"
}
} ]
}
}
}

But, when I try to do the equivalent for a BooleanFilter, multiple
must-terms collide and only one of them is preserved in the
Json-representation, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(constantScoreQuery(boolFilter().must(termFilter("f1",

"v1")).must(termFilter("f2", "v2"))));

s.toString returns:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : {
"term" : {
"f1" : "v1"
}
},
"must" : {
"term" : {
"f2" : "v2"
}
}
}
}
}
}
}

Here both must-Terms within the Hash collide.

Checking the source, I found that the implementation of both classes
differ substantially, so that I suspect there is a misunderstanding on my
side.

Any help is highly appreciated!

Thank you very much.
Björn Hachmann

--

7

Both are exactly the same, I double checked with your query and see that a
proper bool filter is used with 2 should clauses (with the non array
variant). If you think it does not work, can you provide a full curl
recreation, including inserting some sample data?

On Monday, March 19, 2012 1:21:23 PM UTC+1, Hachmaninow wrote:

Hey everybody,

during my experiments with elasticsearch I've found differing behaviour
using the Java-API-Builders for BoolFilters and BoolQueries.

When I try to add multiple must clauses to both of them, the BooleanQuery
seemingly works correct in merging multiple must-terms into an array, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(boolQuery().must(termQuery("f1",

"v1")).must(termQuery("f2", "v2")));

s.toString returns:
{
"query" : {
"bool" : {
"must" : [ {
"term" : {
"f1" : "v1"
}
}, {
"term" : {
"f2" : "v2"
}
} ]
}
}
}

But, when I try to do the equivalent for a BooleanFilter, multiple
must-terms collide and only one of them is preserved in the
Json-representation, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(constantScoreQuery(boolFilter().must(termFilter("f1",

"v1")).must(termFilter("f2", "v2"))));

s.toString returns:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : {
"term" : {
"f1" : "v1"
}
},
"must" : {
"term" : {
"f2" : "v2"
}
}
}
}
}
}
}

Here both must-Terms within the Hash collide.

Checking the source, I found that the implementation of both classes
differ substantially, so that I suspect there is a misunderstanding on my
side.

Any help is highly appreciated!

Thank you very much.
Björn Hachmann

--

8

Resurrects this old thread. The behavior of the boolFilterBuilder causes
trouble - not for the expected results when querying - but e.g when trying
to unit-test using string-representations of the JSon.

I would like to test my results of creating the SearchSourceBuilder like
this: assertJson( "contentTypeFilter_result.json", searchSourceBuilder );

Where the "assertJson"-method uses the
http://jackson.codehaus.org/JSon-parser to create JSonNodes of the
file and the SearchSourcebuilder,
and then comparing. The problem is that, like e.g when verifying the Json
at http://jsonlint.com/, the first element is cut:

{
"query" : {
"query_string" : {
"query" : "test",
"lenient" : true
}
},
"filter" : {
"bool" : {
"must" : {
"terms" : {
"contentType" : [ "contenttypes:mycontenttype1",
"contenttypes:mycontenttype2" ]
}
},
"must" : {
"terms" : {
"space" : [ "myspace2", "myspace1" ]
}
}
}
}
}

will end up like:

{
"query": {
"query_string": {
"query": "test",
"lenient": true
}
},
"filter": {
"bool": {
"must": {
"terms": {
"space": [
"myspace2",
"myspace1"
]
}
}
}
}
}

Im not sure if its the different parsers out there that are wrong and
threats valid JSon wrongly, but anyhow this causes some problems when
working with the BoolFilterBuilder.

--
regards
Runar Myklebust

On Fri, Sep 28, 2012 at 10:08 AM, kimchy kimchy@gmail.com wrote:

Both are exactly the same, I double checked with your query and see that a
proper bool filter is used with 2 should clauses (with the non array
variant). If you think it does not work, can you provide a full curl
recreation, including inserting some sample data?

On Monday, March 19, 2012 1:21:23 PM UTC+1, Hachmaninow wrote:

Hey everybody,

during my experiments with elasticsearch I've found differing behaviour
using the Java-API-Builders for BoolFilters and BoolQueries.

When I try to add multiple must clauses to both of them, the BooleanQuery
seemingly works correct in merging multiple must-terms into an array, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(boolQuery().must(**termQuery("f1",

"v1")).must(termQuery("f2", "v2")));

s.toString returns:
{
"query" : {
"bool" : {
"must" : [ {
"term" : {
"f1" : "v1"
}
}, {
"term" : {
"f2" : "v2"
}
} ]
}
}
}

But, when I try to do the equivalent for a BooleanFilter, multiple
must-terms collide and only one of them is preserved in the
Json-representation, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(constantScoreQuery(**boolFilter().must(termFilter("**f1",

"v1")).must(termFilter("f2", "v2"))));

s.toString returns:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : {
"term" : {
"f1" : "v1"
}
},
"must" : {
"term" : {
"f2" : "v2"
}
}
}
}
}
}
}

Here both must-Terms within the Hash collide.

Checking the source, I found that the implementation of both classes
differ substantially, so that I suspect there is a misunderstanding on my
side.

Any help is highly appreciated!

Thank you very much.
Björn Hachmann

--

--
mvh

Runar Myklebust

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

To answer the last sentence myself, according to the JSON RFC at
http://www.ietf.org/rfc/rfc4627.txt, keys within object SHOULD have unique
names, but no MUST then.

On Thu, Apr 4, 2013 at 2:45 PM, Runar Myklebust runar.a.m@gmail.com wrote:

Resurrects this old thread. The behavior of the boolFilterBuilder causes
trouble - not for the expected results when querying - but e.g when trying
to unit-test using string-representations of the JSon.

I would like to test my results of creating the SearchSourceBuilder like
this: assertJson( "contentTypeFilter_result.json", searchSourceBuilder );

Where the "assertJson"-method uses the http://jackson.codehaus.org/JSon-parser to create JSonNodes of the file and the SearchSourcebuilder,
and then comparing. The problem is that, like e.g when verifying the Json
at http://jsonlint.com/, the first element is cut:

{
"query" : {
"query_string" : {
"query" : "test",
"lenient" : true
}
},
"filter" : {
"bool" : {
"must" : {
"terms" : {
"contentType" : [ "contenttypes:mycontenttype1",
"contenttypes:mycontenttype2" ]
}
},
"must" : {
"terms" : {
"space" : [ "myspace2", "myspace1" ]
}
}
}
}
}

will end up like:

{
"query": {
"query_string": {
"query": "test",
"lenient": true
}
},
"filter": {
"bool": {
"must": {
"terms": {
"space": [
"myspace2",
"myspace1"
]
}
}
}
}
}

Im not sure if its the different parsers out there that are wrong and
threats valid JSon wrongly, but anyhow this causes some problems when
working with the BoolFilterBuilder.

--
regards
Runar Myklebust

On Fri, Sep 28, 2012 at 10:08 AM, kimchy kimchy@gmail.com wrote:

Both are exactly the same, I double checked with your query and see that
a proper bool filter is used with 2 should clauses (with the non array
variant). If you think it does not work, can you provide a full curl
recreation, including inserting some sample data?

On Monday, March 19, 2012 1:21:23 PM UTC+1, Hachmaninow wrote:

Hey everybody,

during my experiments with elasticsearch I've found differing behaviour
using the Java-API-Builders for BoolFilters and BoolQueries.

When I try to add multiple must clauses to both of them, the
BooleanQuery seemingly works correct in merging multiple must-terms into an
array, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(boolQuery().must(**termQuery("f1",

"v1")).must(termQuery("f2", "v2")));

s.toString returns:
{
"query" : {
"bool" : {
"must" : [ {
"term" : {
"f1" : "v1"
}
}, {
"term" : {
"f2" : "v2"
}
} ]
}
}
}

But, when I try to do the equivalent for a BooleanFilter, multiple
must-terms collide and only one of them is preserved in the
Json-representation, eg:

    final SearchRequestBuilder s = client.prepareSearch();
    s.setQuery(constantScoreQuery(**boolFilter().must(termFilter("**f1",

"v1")).must(termFilter("f2", "v2"))));

s.toString returns:
{
"query" : {
"constant_score" : {
"filter" : {
"bool" : {
"must" : {
"term" : {
"f1" : "v1"
}
},
"must" : {
"term" : {
"f2" : "v2"
}
}
}
}
}
}
}

Here both must-Terms within the Hash collide.

Checking the source, I found that the implementation of both classes
differ substantially, so that I suspect there is a misunderstanding on my
side.

Any help is highly appreciated!

Thank you very much.
Björn Hachmann

--

--
mvh

Runar Myklebust

--
mvh

Runar Myklebust

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.