How can I pass a JSON query string to a Query builder

Robert_Simmons · March 23, 2012, 8:06pm

I have been trying for hours to get this working. Basically I have a class
that performs a filtered query and I want the user to be able to pass a
JSON query string to my constructor and then use that on the filtered query
to further reduce the results. I have tried all manner of QueryBuilders
methods and SearchRequestBuilder methods.

Here is the code I settled on but it doesn't allow custom query:

final SearchRequestBuilder srb =
client.prepareSearch(indices).setSearchType(
SearchType.QUERY_AND_FETCH).setTypes(indexTypes);
final GeoDistanceFilterBuilder filter =
FilterBuilders.geoDistanceFilter("point").distance(maxDistance, units)

.lat(address.coordinate.latitude).lon(address.coordinate.longitude)
.geoDistance(GeoDistance.ARC);
final GeoDistanceSortBuilder sort =
SortBuilders.geoDistanceSort("location.point").geoDistance(
GeoDistance.PLANE).point(address.coordinate.latitude,
address.coordinate.longitude);
srb.setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),
filter));
srb.addSort(sort);

What I would like is something like:

final SearchRequestBuilder srb =
client.prepareSearch(indices).setSearchType(
SearchType.QUERY_AND_FETCH).setTypes(indexTypes);
final GeoDistanceFilterBuilder filter =
FilterBuilders.geoDistanceFilter("point").distance(maxDistance, units)

.lat(address.coordinate.latitude).lon(address.coordinate.longitude)
.geoDistance(GeoDistance.ARC);
final GeoDistanceSortBuilder sort =
SortBuilders.geoDistanceSort("location.point").geoDistance(
GeoDistance.PLANE).point(address.coordinate.latitude,
address.coordinate.longitude);
final QueryBuilder query;
if (this.queryJSON == null) {
query = QueryBuilders.fromJSON(this.queryJSON); <<=== doesn't exist.
} else {
query = QueryBuilders.matchAllQuery()
}
srb.setQuery(QueryBuilders.filteredQuery(query, filter));
srb.addSort(sort);

I have tried using srb.setSource() but it completely mangles the JSON
putting a comma just inside the first { and not after the query clause. I
have tried other methods in search request builder but with no success.
Using QueryBuilders.wrapperQuerry(this.queryJSON) doesn't work either.

I have spent hours working on this and would appreciate any pointers.

Lar_Mader · March 23, 2012, 10:51pm

I don't know if this is the best answer (or even helps), but you could
construct your entire query as JSON, and then use a SearchRequest to
execute the query passing in the full JSON query:

client.search(Requests.searchRequest(index).source(jsonStr)).actionGet()

On Fri, Mar 23, 2012 at 1:06 PM, Robert Simmons kraythe@gmail.com wrote:

I have been trying for hours to get this working. Basically I have a class
that performs a filtered query and I want the user to be able to pass a
JSON query string to my constructor and then use that on the filtered query
to further reduce the results. I have tried all manner of QueryBuilders
methods and SearchRequestBuilder methods.

Here is the code I settled on but it doesn't allow custom query:

final SearchRequestBuilder srb =
client.prepareSearch(indices).setSearchType(
SearchType.QUERY_AND_FETCH).setTypes(indexTypes);
final GeoDistanceFilterBuilder filter =
FilterBuilders.geoDistanceFilter("point").distance(maxDistance, units)

.lat(address.coordinate.latitude).lon(address.coordinate.longitude)
.geoDistance(GeoDistance.ARC);
final GeoDistanceSortBuilder sort =
SortBuilders.geoDistanceSort("location.point").geoDistance(
GeoDistance.PLANE).point(address.coordinate.latitude,
address.coordinate.longitude);
srb.setQuery(QueryBuilders.filteredQuery(QueryBuilders.matchAllQuery(),
filter));
srb.addSort(sort);

What I would like is something like:

final SearchRequestBuilder srb =
client.prepareSearch(indices).setSearchType(
SearchType.QUERY_AND_FETCH).setTypes(indexTypes);
final GeoDistanceFilterBuilder filter =
FilterBuilders.geoDistanceFilter("point").distance(maxDistance, units)

.lat(address.coordinate.latitude).lon(address.coordinate.longitude)
.geoDistance(GeoDistance.ARC);
final GeoDistanceSortBuilder sort =
SortBuilders.geoDistanceSort("location.point").geoDistance(
GeoDistance.PLANE).point(address.coordinate.latitude,
address.coordinate.longitude);
final QueryBuilder query;
if (this.queryJSON == null) {
query = QueryBuilders.fromJSON(this.queryJSON); <<=== doesn't exist.
} else {
query = QueryBuilders.matchAllQuery()
}
srb.setQuery(QueryBuilders.filteredQuery(query, filter));
srb.addSort(sort);

I have tried using srb.setSource() but it completely mangles the JSON
putting a comma just inside the first { and not after the query clause. I
have tried other methods in search request builder but with no success.
Using QueryBuilders.wrapperQuerry(this.queryJSON) doesn't work either.

I have spent hours working on this and would appreciate any pointers.

Robert_Simmons · March 23, 2012, 10:59pm

Alas no because I need to combine the passed in string of json with the search filter and sort. Your method would require me to define the entire query. This is analogous to asking a user to pass you a where clause in an RDBMS app.

Igor_Motov · March 24, 2012, 1:00am

I think QueryBuilders.wrapperQuery(jsonStr) is exactly what you want. Can
you post some details why it doesn't work for you? Posting test jsonStr and
the error that you are getting might be helpful.

On Friday, March 23, 2012 6:59:21 PM UTC-4, kraythe wrote:

Alas no because I need to combine the passed in string of json with the
search filter and sort. Your method would require me to define the entire
query. This is analogous to asking a user to pass you a where clause in an
RDBMS app.

Robert_Simmons · March 24, 2012, 2:08am

here is the relevant changes for the wrapper query:

    srb.setQuery(QueryBuilders.filteredQuery(*QueryBuilders.wrapperQuery("

"match_all": {}")*, filter));
srb.addSort(sort);
// -- Now execute the search
final ActionFuture results = srb.execute();

The produced JSON from SRB is:

{
"query" : {
"filtered" : {
"query" : {

```
   "wrapper" : {*
```

     "query" : "ICJtYXRjaF9hbGwiOiB7fQ=="*

   }*
},
"filter" : {
  "geo_distance" : {
    "point" : [ -81.480298, 41.107519 ],
    "distance" : "200.0mi",
    "distance_type" : "arc"
  }
}

}
},
"sort" : [ {
"_geo_distance" : {
"location.point" : [ -81.480298, 41.107519 ],
"distance_type" : "plane"
}
} ]
}

note the garbage inside the query. That is obviously a problem.

So that is why wrapper query doesn't work.

On Friday, March 23, 2012 7:00:55 PM UTC-6, Igor Motov wrote:

I think QueryBuilders.wrapperQuery(jsonStr) is exactly what you want. Can
you post some details why it doesn't work for you? Posting test jsonStr and
the error that you are getting might be helpful.

On Friday, March 23, 2012 6:59:21 PM UTC-4, kraythe wrote:

Alas no because I need to combine the passed in string of json with the
search filter and sort. Your method would require me to define the entire
query. This is analogous to asking a user to pass you a where clause in an
RDBMS app.

Igor_Motov · March 24, 2012, 2:23am

ICJtYXRjaF9hbGwiOiB7fQ== is base64-encoded string " "match_all": {}".
However, it needs additional { ... } to work. Try this:

srb.setQuery(QueryBuilders.filteredQuery(QueryBuilders.wrapperQuery("{"match_all":
{}}"), filter));

On Friday, March 23, 2012 10:08:57 PM UTC-4, kraythe wrote:

here is the relevant changes for the wrapper query:
    srb.setQuery(QueryBuilders.filteredQuery(*QueryBuilders.wrapperQuery(" 
"match_all": {}")*, filter));
srb.addSort(sort);
// -- Now execute the search
final ActionFuture results = srb.execute();

The produced JSON from SRB is:

{
"query" : {
"filtered" : {
"query" : {
   "wrapper" : {*
     "query" : "ICJtYXRjaF9hbGwiOiB7fQ=="*
   }*
},
"filter" : {
  "geo_distance" : {
    "point" : [ -81.480298, 41.107519 ],
    "distance" : "200.0mi",
    "distance_type" : "arc"
  }
}
}
},
"sort" : [ {
"_geo_distance" : {
"location.point" : [ -81.480298, 41.107519 ],
"distance_type" : "plane"
}
} ]
}
note the garbage inside the query. That is obviously a problem.

So that is why wrapper query doesn't work.

On Friday, March 23, 2012 7:00:55 PM UTC-6, Igor Motov wrote:

I think QueryBuilders.wrapperQuery(jsonStr) is exactly what you want.
Can you post some details why it doesn't work for you? Posting test jsonStr
and the error that you are getting might be helpful.

On Friday, March 23, 2012 6:59:21 PM UTC-4, kraythe wrote:

Alas no because I need to combine the passed in string of json with the
search filter and sort. Your method would require me to define the entire
query. This is analogous to asking a user to pass you a where clause in an
RDBMS app.

Robert_Simmons · March 24, 2012, 5:12am

Ok that actually did work, thanks. However the JSON rendering of the query
leaves much to be desired. It is nearly impossible to read if you try to
copy out the query and paste it into curl or something. Is there a way to
get the output unencoded ?

On Friday, March 23, 2012 8:23:19 PM UTC-6, Igor Motov wrote:

ICJtYXRjaF9hbGwiOiB7fQ== is base64-encoded string " "match_all": {}".
However, it needs additional { ... } to work. Try this:

srb.setQuery(QueryBuilders.filteredQuery(QueryBuilders.wrapperQuery("{"match_all":
{}}"), filter));

On Friday, March 23, 2012 10:08:57 PM UTC-4, kraythe wrote:
here is the relevant changes for the wrapper query:
    srb.setQuery(QueryBuilders.filteredQuery(*QueryBuilders.wrapperQuery(" 
"match_all": {}")*, filter));
srb.addSort(sort);
// -- Now execute the search
final ActionFuture results = srb.execute();

The produced JSON from SRB is:

{
"query" : {
"filtered" : {
"query" : {
   "wrapper" : {*
     "query" : "ICJtYXRjaF9hbGwiOiB7fQ=="*
   }*
},
"filter" : {
  "geo_distance" : {
    "point" : [ -81.480298, 41.107519 ],
    "distance" : "200.0mi",
    "distance_type" : "arc"
  }
}
}
},
"sort" : [ {
"_geo_distance" : {
"location.point" : [ -81.480298, 41.107519 ],
"distance_type" : "plane"
}
} ]
}
note the garbage inside the query. That is obviously a problem.

So that is why wrapper query doesn't work.

On Friday, March 23, 2012 7:00:55 PM UTC-6, Igor Motov wrote:

I think QueryBuilders.wrapperQuery(jsonStr) is exactly what you want.
Can you post some details why it doesn't work for you? Posting test jsonStr
and the error that you are getting might be helpful.

On Friday, March 23, 2012 6:59:21 PM UTC-4, kraythe wrote:

Alas no because I need to combine the passed in string of json with the
search filter and sort. Your method would require me to define the entire
query. This is analogous to asking a user to pass you a where clause in an
RDBMS app.

Igor_Motov · March 24, 2012, 1:42pm

You could try parsing the query and then creating your own QueryBuilder
that would regenerate this query on the client side, I guess.

public static QueryBuilder fromSource(String source) {
    byte[] data = source.getBytes();
    return fromSource(data, 0, data.length);
}

public static QueryBuilder fromSource(byte[] data, int offset, int

length) {
final Map<String, Object> source =
XContentHelper.convertToMap(data, offset, length, true).v2();
return new BaseQueryBuilder() {
@Override
protected void doXContent(XContentBuilder builder, Params
params) throws IOException {
for (Map.Entry<String, Object> entry : source.entrySet()) {
builder.field(entry.getKey());
Object value = entry.getValue();
if (value == null) {
builder.nullValue();
} else {
builder.value(value);
}
}
}
};
}

On Saturday, March 24, 2012 1:12:21 AM UTC-4, kraythe wrote:

Ok that actually did work, thanks. However the JSON rendering of the query
leaves much to be desired. It is nearly impossible to read if you try to
copy out the query and paste it into curl or something. Is there a way to
get the output unencoded ?

On Friday, March 23, 2012 8:23:19 PM UTC-6, Igor Motov wrote:
ICJtYXRjaF9hbGwiOiB7fQ== is base64-encoded string " "match_all": {}".
However, it needs additional { ... } to work. Try this:

srb.setQuery(QueryBuilders.filteredQuery(QueryBuilders.wrapperQuery("{"match_all":
{}}"), filter));

On Friday, March 23, 2012 10:08:57 PM UTC-4, kraythe wrote:
here is the relevant changes for the wrapper query:
    srb.setQuery(QueryBuilders.filteredQuery(*QueryBuilders.wrapperQuery(" 
"match_all": {}")*, filter));
srb.addSort(sort);
// -- Now execute the search
final ActionFuture results = srb.execute();

The produced JSON from SRB is:

{
"query" : {
"filtered" : {
"query" : {
   "wrapper" : {*
     "query" : "ICJtYXRjaF9hbGwiOiB7fQ=="*
   }*
},
"filter" : {
  "geo_distance" : {
    "point" : [ -81.480298, 41.107519 ],
    "distance" : "200.0mi",
    "distance_type" : "arc"
  }
}
}
},
"sort" : [ {
"_geo_distance" : {
"location.point" : [ -81.480298, 41.107519 ],
"distance_type" : "plane"
}
} ]
}
note the garbage inside the query. That is obviously a problem.

So that is why wrapper query doesn't work.

On Friday, March 23, 2012 7:00:55 PM UTC-6, Igor Motov wrote:

I think QueryBuilders.wrapperQuery(jsonStr) is exactly what you
want. Can you post some details why it doesn't work for you? Posting test
jsonStr and the error that you are getting might be helpful.

On Friday, March 23, 2012 6:59:21 PM UTC-4, kraythe wrote:

Alas no because I need to combine the passed in string of json with
the search filter and sort. Your method would require me to define the
entire query. This is analogous to asking a user to pass you a where clause
in an RDBMS app.

kimchy · March 25, 2012, 11:56am

The wrapper query just wraps another binary form of json and is parsed on
the shard level when executing the search. The reason why it works like
that is basically performance, instead of parsing the passed json on the
client side in order to recreate it back into the full enclosing query
structure, it simply passed the binary value as is and then the parsing is
done once, when the search is executed on the shard.

We could potentially add a flag to the wrapper query to do the parsing the
generation on the client side as well, then you will get a nicer "printing"
of it, at the price of performance.

On Sat, Mar 24, 2012 at 7:12 AM, Robert Simmons kraythe@gmail.com wrote:

Ok that actually did work, thanks. However the JSON rendering of the query
leaves much to be desired. It is nearly impossible to read if you try to
copy out the query and paste it into curl or something. Is there a way to
get the output unencoded ?

On Friday, March 23, 2012 8:23:19 PM UTC-6, Igor Motov wrote:
ICJtYXRjaF9hbGwiOiB7fQ== is base64-encoded string " "match_all": {}".
However, it needs additional { ... } to work. Try this:

srb.setQuery(QueryBuilders.filteredQuery(QueryBuilders.wrapperQuery("{"match_all":
{}}"), filter));

On Friday, March 23, 2012 10:08:57 PM UTC-4, kraythe wrote:
here is the relevant changes for the wrapper query:
    srb.setQuery(QueryBuilders.filteredQuery(*QueryBuilders.wrapperQuery("
"match_all": {}")*, filter));
srb.addSort(sort);
// -- Now execute the search
final ActionFuture results = srb.execute();

The produced JSON from SRB is:

{
"query" : {
"filtered" : {
"query" : {
   "wrapper" : {*
     "query" : "ICJtYXRjaF9hbGwiOiB7fQ=="*
   }*
},
"filter" : {
  "geo_distance" : {
    "point" : [ -81.480298, 41.107519 ],
    "distance" : "200.0mi",
    "distance_type" : "arc"
  }
}
}
},
"sort" : [ {
"_geo_distance" : {
"location.point" : [ -81.480298, 41.107519 ],
"distance_type" : "plane"
}
} ]
}
note the garbage inside the query. That is obviously a problem.

So that is why wrapper query doesn't work.

On Friday, March 23, 2012 7:00:55 PM UTC-6, Igor Motov wrote:

I think QueryBuilders.wrapperQuery(jsonStr) is exactly what you
want. Can you post some details why it doesn't work for you? Posting test
jsonStr and the error that you are getting might be helpful.

On Friday, March 23, 2012 6:59:21 PM UTC-4, kraythe wrote:

Alas no because I need to combine the passed in string of json with
the search filter and sort. Your method would require me to define the
entire query. This is analogous to asking a user to pass you a where clause
in an RDBMS app.

Robert_Simmons · March 25, 2012, 4:26pm

I think that would be a great addition shay. It would help out in situations of debugging and tracking. However, what would be better is if the querybuilders were serializable to begin with. Then the user could just pass me a querybuilder and I could send it over the wire to another process.

For context sake I will tell you what we are doing. We have a task using JPPF that is being serialized out to nodes to do a bunch of things including searching in ES. The tasks going out to the grid must be serializable and so this is where I hit the snag. I initially planned that they would send me a querybuilder which I would combine with the filter in the search phase of the task but since the querybuilder wasn't serializable I had to break it into a byte stream of JSON and use the wrapped query. Since I had potentially dozens of queries the Base64 encoding made it impossible to figure out which querry was being run where.

Thanks for the help all.

Oh and Shay, the javadoc needs a lot of help. I would be happy to help out there if you need it and are willing to answer a few thousand questions in the meantime.

kimchy · March 25, 2012, 6:20pm

The query builders could be serialized, but you can just "build them as
bytes" and then use the wrapper option if you want to construct more
complex queries around the serialized versions. The only problem you have
is the "toString" here, but its considerably better when it comes to
performance. Note also, that the toString will use json to serialize the
data, but by default, smile is used for the Java API, so its actually a
binary value without the base64 conversion.

On Sun, Mar 25, 2012 at 6:26 PM, Robert Simmons kraythe@gmail.com wrote:

I think that would be a great addition shay. It would help out in
situations of debugging and tracking. However, what would be better is if
the querybuilders were serializable to begin with. Then the user could just
pass me a querybuilder and I could send it over the wire to another process.

For context sake I will tell you what we are doing. We have a task using
JPPF that is being serialized out to nodes to do a bunch of things
including searching in ES. The tasks going out to the grid must be
serializable and so this is where I hit the snag. I initially planned that
they would send me a querybuilder which I would combine with the filter in
the search phase of the task but since the querybuilder wasn't serializable
I had to break it into a byte stream of JSON and use the wrapped query.
Since I had potentially dozens of queries the Base64 encoding made it
impossible to figure out which querry was being run where.

Thanks for the help all.

Oh and Shay, the javadoc needs a lot of help. I would be happy to help out
there if you need it and are willing to answer a few thousand questions in
the meantime.

Robert_Simmons · March 26, 2012, 3:19pm

I could see instances where making the query builders serializable would
pay off pretty well. You may have to add more match terms to a passed in
query for example and it would make it much easier to use in parallel
processing frameworks and so on. I guess there are workarounds but I would
say that if it isn't that hard to make them serializable it would be a big
win.

But at any rate the docs on those query builders need to be updated and a
lot more examples given.

-- Robert