JAVA API for multiple query cluases


(alichi) #1

Hi mates,
I have been asked to use elasticsearch JAVA API (I personally love its
REST API, but I am not the decision maker here). I have been
struggling with finding a good set of classes/methods to build
comprehensive queries. I have already looked into "BaseQueryBuilder"
and its implementing classes but none allow wiring of multiple query
clauses!

I thought maybe the JAVA API is not there yet in terms of milestones,
product management, resources .. (I believe what the team has
delivered so far is AWESOME, and JAVA API is really just a 2010- last
decade thing). But, like I said I am asked to use JAVA API. So, please
help me to find a solution for this. Either I have to develop a JAVA
REST wrapper around the JSON REST API or find a way to use the current
JAVA API to build comprehensive queries.

Any help is very much appreciated! :wink:
Thanks,
Ali


(K.B.) #2

@Ali

BaseQueryBuilder is the foundation, the sub-classes give the details.
If you want to wire, then you misunderstand the way how search with
facets etc. works. What you want is not a chaining of queries, but
instead a filtering of the set. This filtering is important as it
narrows the resultset where the search is applied onto.

In java-api one usually starts with getting an SearchRequestBuilder,
then puts an (only one) query to it and then adds filters to it.

Simply said something like this:

private SearchRequestBuilder initBuilder(SearchRequestBuilder srq) {

    QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

    //boosting
    for (Map.Entry<String, Float> e : boostFields.entrySet()) {
        query.field(e.getKey(), e.getValue());
    }

    srq.setQuery(query); <- only this is the Query!!!
    srq.addFields(fields.toArray(new String[fields.size()]));

    //sorting
    for (Map.Entry<String, SortOrder> e : sortMap.entrySet()) {
        srq.addSort(e.getKey(), e.getValue());
    }

    //filtering <- this is what you want
    AndFilterBuilder myFilters = FilterBuilders.andFilter();
    if (termFilters.size() > 0) {
        for (Map.Entry<String, String> e : termFilters.entrySet())

{
myFilters.add(FilterBuilders.termFilter(e.getKey(),
e.getValue()));
}
srq.setFilter(myFilters);
}

    for (String facet : termFacets.keySet()) {
        TermsFacetBuilder tf =

FacetBuilders.termsFacet(facet).field(facet).size(termFacets.get(facet)).order(getTermsComparator());
if (termFilters.size() > 0) {
tf.facetFilter(myFilters);
}
srq.addFacet(tf);
}

    return srq;
}

please note that this is just an example from my main usage within an
wicket-dataprovider, but it should you show how to tighten things. Be
also sure to cleanse any data you pass to ES prior giving it to the
query:

/**
* function to escape the string from bad chars for the search
*
* @param str the String that should be escaped
* @return an escaped String
*/
@SuppressWarnings({"ConstantConditions"})
public static String smartEscapeQuery(String str) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < str.length(); i++) {
char c = str.charAt(i);
if (c == '\' || c == '+' || c == '-' || c == '!' || c ==
'(' || c == ')' || c == ':'
|| c == '^' || c == '[' || c == ']' || c == '"'
|| c == '{' || c == '}' || c == '~'
|| c == '?' || c == '|' || c == '&' || c == ';'
|| (!Character.isSpaceChar(c) &&
Character.isWhitespace(c))) {
sb.append('\');
}
sb.append(c);
}
return sb.toString();
}

Best,

KB

PS: don't even think of putting a own wrapper around ES<->Java as you
will not even come close to its speed and stability (ES JavaApi joins
the es node network, meaning full clustering, optimized data and
request traveling etc;)

On 21 Nov., 22:20, Ali loghm...@gmail.com wrote:

Hi mates,
I have been asked to use elasticsearch JAVA API (I personally love its
REST API, but I am not the decision maker here). I have been
struggling with finding a good set of classes/methods to build
comprehensive queries. I have already looked into "BaseQueryBuilder"
and its implementing classes but none allow wiring of multiple query
clauses!

I thought maybe the JAVA API is not there yet in terms of milestones,
product management, resources .. (I believe what the team has
delivered so far is AWESOME, and JAVA API is really just a 2010- last
decade thing). But, like I said I am asked to use JAVA API. So, please
help me to find a solution for this. Either I have to develop a JAVA
REST wrapper around the JSON REST API or find a way to use the current
JAVA API to build comprehensive queries.

Any help is very much appreciated! :wink:
Thanks,
Ali


(alichi) #3

@K.B.
Thanks for the reply. You helped a lot with the JAVA API. But I think
the JSON REST API is much cleaner that the convoluted JAVA API. BTW,
by a JAVA wrapper I basically meant a REST layer to generate JSON REST
commands exposing JAVA API to high level ES consumers, it works just
as a facade for commands. This layer has nothing to do with the
underlying ES features like speed and stability unless you tell me
that ES JAVA API is faster than the JSON API which I think cannot be
true. I have to admit that I am new to ES but my understanding so far
is that the JAVA API ultimately translates its command to JSON.

Anyhow, I would like to thank you for the response again.

Best,
Ali

On Nov 21, 4:20 pm, "K.B." korbinian.ba...@googlemail.com wrote:

@Ali

BaseQueryBuilder is the foundation, the sub-classes give the details.
If you want to wire, then you misunderstand the way how search with
facets etc. works. What you want is not a chaining of queries, but
instead a filtering of the set. This filtering is important as it
narrows the resultset where the search is applied onto.

In java-api one usually starts with getting an SearchRequestBuilder,
then puts an (only one) query to it and then adds filters to it.

Simply said something like this:

private SearchRequestBuilder initBuilder(SearchRequestBuilder srq) {

    QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

    //boosting
    for (Map.Entry<String, Float> e : boostFields.entrySet()) {
        query.field(e.getKey(), e.getValue());
    }

    srq.setQuery(query); <- only this is the Query!!!
    srq.addFields(fields.toArray(new String[fields.size()]));

    //sorting
    for (Map.Entry<String, SortOrder> e : sortMap.entrySet()) {
        srq.addSort(e.getKey(), e.getValue());
    }

    //filtering <- this is what you want
    AndFilterBuilder myFilters = FilterBuilders.andFilter();
    if (termFilters.size() > 0) {
        for (Map.Entry<String, String> e : termFilters.entrySet())

{
myFilters.add(FilterBuilders.termFilter(e.getKey(),
e.getValue()));
}
srq.setFilter(myFilters);
}

    for (String facet : termFacets.keySet()) {
        TermsFacetBuilder tf =

FacetBuilders.termsFacet(facet).field(facet).size(termFacets.get(facet)).order(getTermsComparator());
if (termFilters.size() > 0) {
tf.facetFilter(myFilters);
}
srq.addFacet(tf);
}

    return srq;
}

please note that this is just an example from my main usage within an
wicket-dataprovider, but it should you show how to tighten things. Be
also sure to cleanse any data you pass to ES prior giving it to the
query:

/**
* function to escape the string from bad chars for the search
*
* @param str the String that should be escaped
* @return an escaped String
*/
@SuppressWarnings({"ConstantConditions"})
public static String smartEscapeQuery(String str) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < str.length(); i++) {
char c = str.charAt(i);
if (c == '\' || c == '+' || c == '-' || c == '!' || c ==
'(' || c == ')' || c == ':'
|| c == '^' || c == '[' || c == ']' || c == '"'
|| c == '{' || c == '}' || c == '~'
|| c == '?' || c == '|' || c == '&' || c == ';'
|| (!Character.isSpaceChar(c) &&
Character.isWhitespace(c))) {
sb.append('\');
}
sb.append(c);
}
return sb.toString();
}

Best,

KB

PS: don't even think of putting a own wrapper around ES<->Java as you
will not even come close to its speed and stability (ES JavaApi joins
the es node network, meaning full clustering, optimized data and
request traveling etc;)

On 21 Nov., 22:20, Ali loghm...@gmail.com wrote:

Hi mates,
I have been asked to use elasticsearch JAVA API (I personally love its
REST API, but I am not the decision maker here). I have been
struggling with finding a good set of classes/methods to build
comprehensive queries. I have already looked into "BaseQueryBuilder"
and its implementing classes but none allow wiring of multiple query
clauses!

I thought maybe the JAVA API is not there yet in terms of milestones,
product management, resources .. (I believe what the team has
delivered so far is AWESOME, and JAVA API is really just a 2010- last
decade thing). But, like I said I am asked to use JAVA API. So, please
help me to find a solution for this. Either I have to develop a JAVA
REST wrapper around the JSON REST API or find a way to use the current
JAVA API to build comprehensive queries.

Any help is very much appreciated! :wink:
Thanks,
Ali


(alichi) #4

@ ES gurus,
I am still a little bit baffled with the API: How can I make a clean
query without defining a query around a value and then wire filters to
it! The code below (a more complete version of it can be found above)

QueryStringQueryBuilder query =
QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);
.....
and then adding filters to this query (In the first place why do I
need to set that querystring up there!)
.....
I just do not understand why one would need to first crate a query
with a value and then add filters to it! I need to build a query with
multiple clauses over several fields with the ability to set different
values. boosting, analyzer and etc for each field at once! I can do
this with Lucene! I should be able to do the same with ES too.

I would rather look for something like this:

QueryBuilders.clause(value).fields(values...,
boost...).anaylzer(....).operator(...).clause(value).fields(.....).anaylzer(....);

Thanks,
Ali

On Nov 21, 5:01 pm, Ali loghm...@gmail.com wrote:

@K.B.
Thanks for the reply. You helped a lot with the JAVA API. But I think
the JSON REST API is much cleaner that the convoluted JAVA API. BTW,
by a JAVA wrapper I basically meant a REST layer to generate JSON REST
commands exposing JAVA API to high level ES consumers, it works just
as a facade for commands. This layer has nothing to do with the
underlying ES features like speed and stability unless you tell me
that ES JAVA API is faster than the JSON API which I think cannot be
true. I have to admit that I am new to ES but my understanding so far
is that the JAVA API ultimately translates its command to JSON.

Anyhow, I would like to thank you for the response again.

Best,
Ali

On Nov 21, 4:20 pm, "K.B." korbinian.ba...@googlemail.com wrote:

@Ali

BaseQueryBuilder is the foundation, the sub-classes give the details.
If you want to wire, then you misunderstand the way how search with
facets etc. works. What you want is not a chaining of queries, but
instead a filtering of the set. This filtering is important as it
narrows the resultset where the search is applied onto.

In java-api one usually starts with getting an SearchRequestBuilder,
then puts an (only one) query to it and then adds filters to it.

Simply said something like this:

private SearchRequestBuilder initBuilder(SearchRequestBuilder srq) {

    QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

    //boosting
    for (Map.Entry<String, Float> e : boostFields.entrySet()) {
        query.field(e.getKey(), e.getValue());
    }
    srq.setQuery(query); <- only this is the Query!!!
    srq.addFields(fields.toArray(new String[fields.size()]));
    //sorting
    for (Map.Entry<String, SortOrder> e : sortMap.entrySet()) {
        srq.addSort(e.getKey(), e.getValue());
    }
    //filtering <- this is what you want
    AndFilterBuilder myFilters = FilterBuilders.andFilter();
    if (termFilters.size() > 0) {
        for (Map.Entry<String, String> e : termFilters.entrySet())

{
myFilters.add(FilterBuilders.termFilter(e.getKey(),
e.getValue()));
}
srq.setFilter(myFilters);
}

    for (String facet : termFacets.keySet()) {
        TermsFacetBuilder tf =

FacetBuilders.termsFacet(facet).field(facet).size(termFacets.get(facet)).order(getTermsComparator());
if (termFilters.size() > 0) {
tf.facetFilter(myFilters);
}
srq.addFacet(tf);
}

    return srq;
}

please note that this is just an example from my main usage within an
wicket-dataprovider, but it should you show how to tighten things. Be
also sure to cleanse any data you pass to ES prior giving it to the
query:

/**
* function to escape the string from bad chars for the search
*
* @param str the String that should be escaped
* @return an escaped String
*/
@SuppressWarnings({"ConstantConditions"})
public static String smartEscapeQuery(String str) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < str.length(); i++) {
char c = str.charAt(i);
if (c == '\' || c == '+' || c == '-' || c == '!' || c ==
'(' || c == ')' || c == ':'
|| c == '^' || c == '[' || c == ']' || c == '"'
|| c == '{' || c == '}' || c == '~'
|| c == '?' || c == '|' || c == '&' || c == ';'
|| (!Character.isSpaceChar(c) &&
Character.isWhitespace(c))) {
sb.append('\');
}
sb.append(c);
}
return sb.toString();
}

Best,

KB

PS: don't even think of putting a own wrapper around ES<->Java as you
will not even come close to its speed and stability (ES JavaApi joins
the es node network, meaning full clustering, optimized data and
request traveling etc;)

On 21 Nov., 22:20, Ali loghm...@gmail.com wrote:

Hi mates,
I have been asked to use elasticsearch JAVA API (I personally love its
REST API, but I am not the decision maker here). I have been
struggling with finding a good set of classes/methods to build
comprehensive queries. I have already looked into "BaseQueryBuilder"
and its implementing classes but none allow wiring of multiple query
clauses!

I thought maybe the JAVA API is not there yet in terms of milestones,
product management, resources .. (I believe what the team has
delivered so far is AWESOME, and JAVA API is really just a 2010- last
decade thing). But, like I said I am asked to use JAVA API. So, please
help me to find a solution for this. Either I have to develop a JAVA
REST wrapper around the JSON REST API or find a way to use the current
JAVA API to build comprehensive queries.

Any help is very much appreciated! :wink:
Thanks,
Ali


(Shay Banon) #5

If you can do it with Lucene, then it shouldn't be difficult for you to use
elasticsearch. The Java API has builders for all the relevant queries that
are supported. If you want to use Lucene FilteredQuery, then using
QueryBuilders.filteredQuery(query, filter). If you want to have a
BooleanQuery, use QueryBuilders.boolQuery, which you can then add
must/mustNot/should clauses to.

The QueryBuilders factory class for queries and FilterBuilders factory
class for filters are a simple starting point to use it.

On Tue, Nov 22, 2011 at 5:12 AM, Ali loghmani@gmail.com wrote:

@ ES gurus,
I am still a little bit baffled with the API: How can I make a clean
query without defining a query around a value and then wire filters to
it! The code below (a more complete version of it can be found above)

QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);
.....
and then adding filters to this query (In the first place why do I
need to set that querystring up there!)
.....
I just do not understand why one would need to first crate a query
with a value and then add filters to it! I need to build a query with
multiple clauses over several fields with the ability to set different
values. boosting, analyzer and etc for each field at once! I can do
this with Lucene! I should be able to do the same with ES too.

I would rather look for something like this:

QueryBuilders.clause(value).fields(values...,

boost...).anaylzer(....).operator(...).clause(value).fields(.....).anaylzer(....);

Thanks,
Ali

On Nov 21, 5:01 pm, Ali loghm...@gmail.com wrote:

@K.B.
Thanks for the reply. You helped a lot with the JAVA API. But I think
the JSON REST API is much cleaner that the convoluted JAVA API. BTW,
by a JAVA wrapper I basically meant a REST layer to generate JSON REST
commands exposing JAVA API to high level ES consumers, it works just
as a facade for commands. This layer has nothing to do with the
underlying ES features like speed and stability unless you tell me
that ES JAVA API is faster than the JSON API which I think cannot be
true. I have to admit that I am new to ES but my understanding so far
is that the JAVA API ultimately translates its command to JSON.

Anyhow, I would like to thank you for the response again.

Best,
Ali

On Nov 21, 4:20 pm, "K.B." korbinian.ba...@googlemail.com wrote:

@Ali

BaseQueryBuilder is the foundation, the sub-classes give the details.
If you want to wire, then you misunderstand the way how search with
facets etc. works. What you want is not a chaining of queries, but
instead a filtering of the set. This filtering is important as it
narrows the resultset where the search is applied onto.

In java-api one usually starts with getting an SearchRequestBuilder,
then puts an (only one) query to it and then adds filters to it.

Simply said something like this:

private SearchRequestBuilder initBuilder(SearchRequestBuilder srq) {

    QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

    //boosting
    for (Map.Entry<String, Float> e : boostFields.entrySet()) {
        query.field(e.getKey(), e.getValue());
    }
    srq.setQuery(query); <- only this is the Query!!!
    srq.addFields(fields.toArray(new String[fields.size()]));
    //sorting
    for (Map.Entry<String, SortOrder> e : sortMap.entrySet()) {
        srq.addSort(e.getKey(), e.getValue());
    }
    //filtering <- this is what you want
    AndFilterBuilder myFilters = FilterBuilders.andFilter();
    if (termFilters.size() > 0) {
        for (Map.Entry<String, String> e : termFilters.entrySet())

{
myFilters.add(FilterBuilders.termFilter(e.getKey(),
e.getValue()));
}
srq.setFilter(myFilters);
}

    for (String facet : termFacets.keySet()) {
        TermsFacetBuilder tf =

FacetBuilders.termsFacet(facet).field(facet).size(termFacets.get(facet)).order(getTermsComparator());

        if (termFilters.size() > 0) {
            tf.facetFilter(myFilters);
        }
        srq.addFacet(tf);
    }
    return srq;
}

please note that this is just an example from my main usage within an
wicket-dataprovider, but it should you show how to tighten things. Be
also sure to cleanse any data you pass to ES prior giving it to the
query:

/**
* function to escape the string from bad chars for the search
*
* @param str the String that should be escaped
* @return an escaped String
*/
@SuppressWarnings({"ConstantConditions"})
public static String smartEscapeQuery(String str) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < str.length(); i++) {
char c = str.charAt(i);
if (c == '\' || c == '+' || c == '-' || c == '!' || c ==
'(' || c == ')' || c == ':'
|| c == '^' || c == '[' || c == ']' || c == '"'
|| c == '{' || c == '}' || c == '~'
|| c == '?' || c == '|' || c == '&' || c == ';'
|| (!Character.isSpaceChar(c) &&
Character.isWhitespace(c))) {
sb.append('\');
}
sb.append(c);
}
return sb.toString();
}

Best,

KB

PS: don't even think of putting a own wrapper around ES<->Java as you
will not even come close to its speed and stability (ES JavaApi joins
the es node network, meaning full clustering, optimized data and
request traveling etc;)

On 21 Nov., 22:20, Ali loghm...@gmail.com wrote:

Hi mates,
I have been asked to use elasticsearch JAVA API (I personally love
its

REST API, but I am not the decision maker here). I have been
struggling with finding a good set of classes/methods to build
comprehensive queries. I have already looked into "BaseQueryBuilder"
and its implementing classes but none allow wiring of multiple query
clauses!

I thought maybe the JAVA API is not there yet in terms of milestones,
product management, resources .. (I believe what the team has
delivered so far is AWESOME, and JAVA API is really just a 2010- last
decade thing). But, like I said I am asked to use JAVA API. So,
please

help me to find a solution for this. Either I have to develop a JAVA
REST wrapper around the JSON REST API or find a way to use the
current

JAVA API to build comprehensive queries.

Any help is very much appreciated! :wink:
Thanks,
Ali


(alichi) #6

@Shay,
Thanks a bunch, I used to use Lucene query parser instead of the JAVA
API directly but now I understand where the API trend is coming from
in ES. Anyways, thanks for the prompt followup and one last question,
do you recommend using the JAVA API or the JSON REST API ( I am
personally in love with the REST API)?
Thanks,
Ali

On Nov 22, 4:37 am, Shay Banon kim...@gmail.com wrote:

If you can do it with Lucene, then it shouldn't be difficult for you to use
elasticsearch. The Java API has builders for all the relevant queries that
are supported. If you want to use Lucene FilteredQuery, then using
QueryBuilders.filteredQuery(query, filter). If you want to have a
BooleanQuery, use QueryBuilders.boolQuery, which you can then add
must/mustNot/should clauses to.

The QueryBuilders factory class for queries and FilterBuilders factory
class for filters are a simple starting point to use it.

On Tue, Nov 22, 2011 at 5:12 AM, Ali loghm...@gmail.com wrote:

@ ES gurus,
I am still a little bit baffled with the API: How can I make a clean
query without defining a query around a value and then wire filters to
it! The code below (a more complete version of it can be found above)

QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);
.....
and then adding filters to this query (In the first place why do I
need to set that querystring up there!)
.....
I just do not understand why one would need to first crate a query
with a value and then add filters to it! I need to build a query with
multiple clauses over several fields with the ability to set different
values. boosting, analyzer and etc for each field at once! I can do
this with Lucene! I should be able to do the same with ES too.

I would rather look for something like this:

QueryBuilders.clause(value).fields(values...,

boost...).anaylzer(....).operator(...).clause(value).fields(.....).anaylzer(....);

Thanks,
Ali

On Nov 21, 5:01 pm, Ali loghm...@gmail.com wrote:

@K.B.
Thanks for the reply. You helped a lot with the JAVA API. But I think
the JSON REST API is much cleaner that the convoluted JAVA API. BTW,
by a JAVA wrapper I basically meant a REST layer to generate JSON REST
commands exposing JAVA API to high level ES consumers, it works just
as a facade for commands. This layer has nothing to do with the
underlying ES features like speed and stability unless you tell me
that ES JAVA API is faster than the JSON API which I think cannot be
true. I have to admit that I am new to ES but my understanding so far
is that the JAVA API ultimately translates its command to JSON.

Anyhow, I would like to thank you for the response again.

Best,
Ali

On Nov 21, 4:20 pm, "K.B." korbinian.ba...@googlemail.com wrote:

@Ali

BaseQueryBuilder is the foundation, the sub-classes give the details.
If you want to wire, then you misunderstand the way how search with
facets etc. works. What you want is not a chaining of queries, but
instead a filtering of the set. This filtering is important as it
narrows the resultset where the search is applied onto.

In java-api one usually starts with getting an SearchRequestBuilder,
then puts an (only one) query to it and then adds filters to it.

Simply said something like this:

private SearchRequestBuilder initBuilder(SearchRequestBuilder srq) {

    QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

    //boosting
    for (Map.Entry<String, Float> e : boostFields.entrySet()) {
        query.field(e.getKey(), e.getValue());
    }
    srq.setQuery(query); <- only this is the Query!!!
    srq.addFields(fields.toArray(new String[fields.size()]));
    //sorting
    for (Map.Entry<String, SortOrder> e : sortMap.entrySet()) {
        srq.addSort(e.getKey(), e.getValue());
    }
    //filtering <- this is what you want
    AndFilterBuilder myFilters = FilterBuilders.andFilter();
    if (termFilters.size() > 0) {
        for (Map.Entry<String, String> e : termFilters.entrySet())

{
myFilters.add(FilterBuilders.termFilter(e.getKey(),
e.getValue()));
}
srq.setFilter(myFilters);
}

    for (String facet : termFacets.keySet()) {
        TermsFacetBuilder tf =

FacetBuilders.termsFacet(facet).field(facet).size(termFacets.get(facet)).order(getTermsComparator());

        if (termFilters.size() > 0) {
            tf.facetFilter(myFilters);
        }
        srq.addFacet(tf);
    }
    return srq;
}

please note that this is just an example from my main usage within an
wicket-dataprovider, but it should you show how to tighten things. Be
also sure to cleanse any data you pass to ES prior giving it to the
query:

/**
* function to escape the string from bad chars for the search
*
* @param str the String that should be escaped
* @return an escaped String
*/
@SuppressWarnings({"ConstantConditions"})
public static String smartEscapeQuery(String str) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < str.length(); i++) {
char c = str.charAt(i);
if (c == '\' || c == '+' || c == '-' || c == '!' || c ==
'(' || c == ')' || c == ':'
|| c == '^' || c == '[' || c == ']' || c == '"'
|| c == '{' || c == '}' || c == '~'
|| c == '?' || c == '|' || c == '&' || c == ';'
|| (!Character.isSpaceChar(c) &&
Character.isWhitespace(c))) {
sb.append('\');
}
sb.append(c);
}
return sb.toString();
}

Best,

KB

PS: don't even think of putting a own wrapper around ES<->Java as you
will not even come close to its speed and stability (ES JavaApi joins
the es node network, meaning full clustering, optimized data and
request traveling etc;)

On 21 Nov., 22:20, Ali loghm...@gmail.com wrote:

Hi mates,
I have been asked to use elasticsearch JAVA API (I personally love
its

REST API, but I am not the decision maker here). I have been
struggling with finding a good set of classes/methods to build
comprehensive queries. I have already looked into "BaseQueryBuilder"
and its implementing classes but none allow wiring of multiple query
clauses!

I thought maybe the JAVA API is not there yet in terms of milestones,
product management, resources .. (I believe what the team has
delivered so far is AWESOME, and JAVA API is really just a 2010- last
decade thing). But, like I said I am asked to use JAVA API. So,
please

help me to find a solution for this. Either I have to develop a JAVA
REST wrapper around the JSON REST API or find a way to use the
current

JAVA API to build comprehensive queries.

Any help is very much appreciated! :wink:
Thanks,
Ali


(K.B.) #7

Clearly JavaApi, as with the Rest-Api you need 1 ES endpoint to access
meaning a single-point-of-failure as the javaApi joins the ES cluster
and won't be interested if one of the nodes die. Beside that java Api
has better performance as th data-exchange is optimized in terms of
data-streaming etc.;

On 22 Nov., 21:02, Ali loghm...@gmail.com wrote:

@Shay,
Thanks a bunch, I used to use Lucene query parser instead of the JAVA
API directly but now I understand where the API trend is coming from
in ES. Anyways, thanks for the prompt followup and one last question,
do you recommend using the JAVA API or the JSON REST API ( I am
personally in love with the REST API)?
Thanks,
Ali

On Nov 22, 4:37 am, Shay Banon kim...@gmail.com wrote:

If you can do it with Lucene, then it shouldn't be difficult for you to use
elasticsearch. The Java API has builders for all the relevant queries that
are supported. If you want to use Lucene FilteredQuery, then using
QueryBuilders.filteredQuery(query, filter). If you want to have a
BooleanQuery, use QueryBuilders.boolQuery, which you can then add
must/mustNot/should clauses to.

The QueryBuilders factory class for queries and FilterBuilders factory
class for filters are a simple starting point to use it.

On Tue, Nov 22, 2011 at 5:12 AM, Ali loghm...@gmail.com wrote:

@ ES gurus,
I am still a little bit baffled with the API: How can I make a clean
query without defining a query around a value and then wire filters to
it! The code below (a more complete version of it can be found above)

QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);
.....
and then adding filters to this query (In the first place why do I
need to set that querystring up there!)
.....
I just do not understand why one would need to first crate a query
with a value and then add filters to it! I need to build a query with
multiple clauses over several fields with the ability to set different
values. boosting, analyzer and etc for each field at once! I can do
this with Lucene! I should be able to do the same with ES too.

I would rather look for something like this:

QueryBuilders.clause(value).fields(values...,

boost...).anaylzer(....).operator(...).clause(value).fields(.....).anaylzer(....);

Thanks,
Ali

On Nov 21, 5:01 pm, Ali loghm...@gmail.com wrote:

@K.B.
Thanks for the reply. You helped a lot with the JAVA API. But I think
the JSON REST API is much cleaner that the convoluted JAVA API. BTW,
by a JAVA wrapper I basically meant a REST layer to generate JSON REST
commands exposing JAVA API to high level ES consumers, it works just
as a facade for commands. This layer has nothing to do with the
underlying ES features like speed and stability unless you tell me
that ES JAVA API is faster than the JSON API which I think cannot be
true. I have to admit that I am new to ES but my understanding so far
is that the JAVA API ultimately translates its command to JSON.

Anyhow, I would like to thank you for the response again.

Best,
Ali

On Nov 21, 4:20 pm, "K.B." korbinian.ba...@googlemail.com wrote:

@Ali

BaseQueryBuilder is the foundation, the sub-classes give the details.
If you want to wire, then you misunderstand the way how search with
facets etc. works. What you want is not a chaining of queries, but
instead a filtering of the set. This filtering is important as it
narrows the resultset where the search is applied onto.

In java-api one usually starts with getting an SearchRequestBuilder,
then puts an (only one) query to it and then adds filters to it.

Simply said something like this:

private SearchRequestBuilder initBuilder(SearchRequestBuilder srq) {

    QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

    //boosting
    for (Map.Entry<String, Float> e : boostFields.entrySet()) {
        query.field(e.getKey(), e.getValue());
    }
    srq.setQuery(query); <- only this is the Query!!!
    srq.addFields(fields.toArray(new String[fields.size()]));
    //sorting
    for (Map.Entry<String, SortOrder> e : sortMap.entrySet()) {
        srq.addSort(e.getKey(), e.getValue());
    }
    //filtering <- this is what you want
    AndFilterBuilder myFilters = FilterBuilders.andFilter();
    if (termFilters.size() > 0) {
        for (Map.Entry<String, String> e : termFilters.entrySet())

{
myFilters.add(FilterBuilders.termFilter(e.getKey(),
e.getValue()));
}
srq.setFilter(myFilters);
}

    for (String facet : termFacets.keySet()) {
        TermsFacetBuilder tf =

FacetBuilders.termsFacet(facet).field(facet).size(termFacets.get(facet)).order(getTermsComparator());

        if (termFilters.size() > 0) {
            tf.facetFilter(myFilters);
        }
        srq.addFacet(tf);
    }
    return srq;
}

please note that this is just an example from my main usage within an
wicket-dataprovider, but it should you show how to tighten things. Be
also sure to cleanse any data you pass to ES prior giving it to the
query:

/**
* function to escape the string from bad chars for the search
*
* @param str the String that should be escaped
* @return an escaped String
*/
@SuppressWarnings({"ConstantConditions"})
public static String smartEscapeQuery(String str) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < str.length(); i++) {
char c = str.charAt(i);
if (c == '\' || c == '+' || c == '-' || c == '!' || c ==
'(' || c == ')' || c == ':'
|| c == '^' || c == '[' || c == ']' || c == '"'
|| c == '{' || c == '}' || c == '~'
|| c == '?' || c == '|' || c == '&' || c == ';'
|| (!Character.isSpaceChar(c) &&
Character.isWhitespace(c))) {
sb.append('\');
}
sb.append(c);
}
return sb.toString();
}

Best,

KB

PS: don't even think of putting a own wrapper around ES<->Java as you
will not even come close to its speed and stability (ES JavaApi joins
the es node network, meaning full clustering, optimized data and
request traveling etc;)

On 21 Nov., 22:20, Ali loghm...@gmail.com wrote:

Hi mates,
I have been asked to use elasticsearch JAVA API (I personally love
its

REST API, but I am not the decision maker here). I have been
struggling with finding a good set of classes/methods to build
comprehensive queries. I have already looked into "BaseQueryBuilder"
and its implementing classes but none allow wiring of multiple query
clauses!

I thought maybe the JAVA API is not there yet in terms of milestones,
product management, resources .. (I believe what the team has
delivered so far is AWESOME, and JAVA API is really just a 2010- last
decade thing). But, like I said I am asked to use JAVA API. So,
please

help me to find a solution for this. Either I have to develop a JAVA
REST wrapper around the JSON REST API or find a way to use the
current

JAVA API to build comprehensive queries.

Any help is very much appreciated! :wink:
Thanks,
Ali


(Shay Banon) #8

Up to you on Java API vs. REST API, note that the Query DSL ends up
converting to the REST format of a query, just statically typed...

On Tue, Nov 22, 2011 at 10:02 PM, Ali loghmani@gmail.com wrote:

@Shay,
Thanks a bunch, I used to use Lucene query parser instead of the JAVA
API directly but now I understand where the API trend is coming from
in ES. Anyways, thanks for the prompt followup and one last question,
do you recommend using the JAVA API or the JSON REST API ( I am
personally in love with the REST API)?
Thanks,
Ali

On Nov 22, 4:37 am, Shay Banon kim...@gmail.com wrote:

If you can do it with Lucene, then it shouldn't be difficult for you to
use
elasticsearch. The Java API has builders for all the relevant queries
that
are supported. If you want to use Lucene FilteredQuery, then using
QueryBuilders.filteredQuery(query, filter). If you want to have a
BooleanQuery, use QueryBuilders.boolQuery, which you can then add
must/mustNot/should clauses to.

The QueryBuilders factory class for queries and FilterBuilders factory
class for filters are a simple starting point to use it.

On Tue, Nov 22, 2011 at 5:12 AM, Ali loghm...@gmail.com wrote:

@ ES gurus,
I am still a little bit baffled with the API: How can I make a clean
query without defining a query around a value and then wire filters to
it! The code below (a more complete version of it can be found above)

QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

.....
and then adding filters to this query (In the first place why do I
need to set that querystring up there!)
.....
I just do not understand why one would need to first crate a query
with a value and then add filters to it! I need to build a query with
multiple clauses over several fields with the ability to set different
values. boosting, analyzer and etc for each field at once! I can do
this with Lucene! I should be able to do the same with ES too.

I would rather look for something like this:

QueryBuilders.clause(value).fields(values...,

boost...).anaylzer(....).operator(...).clause(value).fields(.....).anaylzer(....);

Thanks,
Ali

On Nov 21, 5:01 pm, Ali loghm...@gmail.com wrote:

@K.B.
Thanks for the reply. You helped a lot with the JAVA API. But I think
the JSON REST API is much cleaner that the convoluted JAVA API. BTW,
by a JAVA wrapper I basically meant a REST layer to generate JSON
REST

commands exposing JAVA API to high level ES consumers, it works just
as a facade for commands. This layer has nothing to do with the
underlying ES features like speed and stability unless you tell me
that ES JAVA API is faster than the JSON API which I think cannot be
true. I have to admit that I am new to ES but my understanding so far
is that the JAVA API ultimately translates its command to JSON.

Anyhow, I would like to thank you for the response again.

Best,
Ali

On Nov 21, 4:20 pm, "K.B." korbinian.ba...@googlemail.com wrote:

@Ali

BaseQueryBuilder is the foundation, the sub-classes give the
details.

If you want to wire, then you misunderstand the way how search with
facets etc. works. What you want is not a chaining of queries, but
instead a filtering of the set. This filtering is important as it
narrows the resultset where the search is applied onto.

In java-api one usually starts with getting an
SearchRequestBuilder,

then puts an (only one) query to it and then adds filters to it.

Simply said something like this:

private SearchRequestBuilder initBuilder(SearchRequestBuilder srq)
{

    QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

    //boosting
    for (Map.Entry<String, Float> e : boostFields.entrySet()) {
        query.field(e.getKey(), e.getValue());
    }
    srq.setQuery(query); <- only this is the Query!!!
    srq.addFields(fields.toArray(new String[fields.size()]));
    //sorting
    for (Map.Entry<String, SortOrder> e : sortMap.entrySet()) {
        srq.addSort(e.getKey(), e.getValue());
    }
    //filtering <- this is what you want
    AndFilterBuilder myFilters = FilterBuilders.andFilter();
    if (termFilters.size() > 0) {
        for (Map.Entry<String, String> e :

termFilters.entrySet())

{
myFilters.add(FilterBuilders.termFilter(e.getKey(),
e.getValue()));
}
srq.setFilter(myFilters);
}

    for (String facet : termFacets.keySet()) {
        TermsFacetBuilder tf =

FacetBuilders.termsFacet(facet).field(facet).size(termFacets.get(facet)).order(getTermsComparator());

        if (termFilters.size() > 0) {
            tf.facetFilter(myFilters);
        }
        srq.addFacet(tf);
    }
    return srq;
}

please note that this is just an example from my main usage within
an

wicket-dataprovider, but it should you show how to tighten things.
Be

also sure to cleanse any data you pass to ES prior giving it to the
query:

/**
* function to escape the string from bad chars for the search
*
* @param str the String that should be escaped
* @return an escaped String
*/
@SuppressWarnings({"ConstantConditions"})
public static String smartEscapeQuery(String str) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < str.length(); i++) {
char c = str.charAt(i);
if (c == '\' || c == '+' || c == '-' || c == '!' || c
==

'(' || c == ')' || c == ':'
|| c == '^' || c == '[' || c == ']' || c ==
'"'

|| c == '{' || c == '}' || c == '~'
|| c == '?' || c == '|' || c == '&' || c == ';'
|| (!Character.isSpaceChar(c) &&
Character.isWhitespace(c))) {
sb.append('\');
}
sb.append(c);
}
return sb.toString();
}

Best,

KB

PS: don't even think of putting a own wrapper around ES<->Java as
you

will not even come close to its speed and stability (ES JavaApi
joins

the es node network, meaning full clustering, optimized data and
request traveling etc;)

On 21 Nov., 22:20, Ali loghm...@gmail.com wrote:

Hi mates,
I have been asked to use elasticsearch JAVA API (I personally
love

its

REST API, but I am not the decision maker here). I have been
struggling with finding a good set of classes/methods to build
comprehensive queries. I have already looked into
"BaseQueryBuilder"

and its implementing classes but none allow wiring of multiple
query

clauses!

I thought maybe the JAVA API is not there yet in terms of
milestones,

product management, resources .. (I believe what the team has
delivered so far is AWESOME, and JAVA API is really just a 2010-
last

decade thing). But, like I said I am asked to use JAVA API. So,
please

help me to find a solution for this. Either I have to develop a
JAVA

REST wrapper around the JSON REST API or find a way to use the
current

JAVA API to build comprehensive queries.

Any help is very much appreciated! :wink:
Thanks,
Ali


(alichi) #9

@Shay Great, that is what I thought.
@K.B.consuming the REST API has nothing to do with single point of
failure. As a matter of fact, when the ES cluster is up and running
you can just fire REST commands to it ... I do not understand why you
think there will be a single point of failure when you will have one
REST call per each request created by your app and responded back by
the ES cluster.
I personally prefer the pluggability aspect of the REST API as well
its inherent loosely coupled scalable aspects. With JAVA API things
become coupled, does not matter how well and loosely coupled you
design your hierarchy of classes, still the REST API is more loosely
coupled. Especially, when you are doing a large project dealing with
legacy and new technologies (PHP, NodeJS, Java, AJAX, etc...) you will
definitely need something easy to use.
Thanks,
Ali

On Nov 23, 2:24 am, Shay Banon kim...@gmail.com wrote:

Up to you on Java API vs. REST API, note that the Query DSL ends up
converting to the REST format of a query, just statically typed...

On Tue, Nov 22, 2011 at 10:02 PM, Ali loghm...@gmail.com wrote:

@Shay,
Thanks a bunch, I used to use Lucene query parser instead of the JAVA
API directly but now I understand where the API trend is coming from
in ES. Anyways, thanks for the prompt followup and one last question,
do you recommend using the JAVA API or the JSON REST API ( I am
personally in love with the REST API)?
Thanks,
Ali

On Nov 22, 4:37 am, Shay Banon kim...@gmail.com wrote:

If you can do it with Lucene, then it shouldn't be difficult for you to
use
elasticsearch. The Java API has builders for all the relevant queries
that
are supported. If you want to use Lucene FilteredQuery, then using
QueryBuilders.filteredQuery(query, filter). If you want to have a
BooleanQuery, use QueryBuilders.boolQuery, which you can then add
must/mustNot/should clauses to.

The QueryBuilders factory class for queries and FilterBuilders factory
class for filters are a simple starting point to use it.

On Tue, Nov 22, 2011 at 5:12 AM, Ali loghm...@gmail.com wrote:

@ ES gurus,
I am still a little bit baffled with the API: How can I make a clean
query without defining a query around a value and then wire filters to
it! The code below (a more complete version of it can be found above)

QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

.....
and then adding filters to this query (In the first place why do I
need to set that querystring up there!)
.....
I just do not understand why one would need to first crate a query
with a value and then add filters to it! I need to build a query with
multiple clauses over several fields with the ability to set different
values. boosting, analyzer and etc for each field at once! I can do
this with Lucene! I should be able to do the same with ES too.

I would rather look for something like this:

QueryBuilders.clause(value).fields(values...,

boost...).anaylzer(....).operator(...).clause(value).fields(.....).anaylzer(....);

Thanks,
Ali

On Nov 21, 5:01 pm, Ali loghm...@gmail.com wrote:

@K.B.
Thanks for the reply. You helped a lot with the JAVA API. But I think
the JSON REST API is much cleaner that the convoluted JAVA API. BTW,
by a JAVA wrapper I basically meant a REST layer to generate JSON
REST

commands exposing JAVA API to high level ES consumers, it works just
as a facade for commands. This layer has nothing to do with the
underlying ES features like speed and stability unless you tell me
that ES JAVA API is faster than the JSON API which I think cannot be
true. I have to admit that I am new to ES but my understanding so far
is that the JAVA API ultimately translates its command to JSON.

Anyhow, I would like to thank you for the response again.

Best,
Ali

On Nov 21, 4:20 pm, "K.B." korbinian.ba...@googlemail.com wrote:

@Ali

BaseQueryBuilder is the foundation, the sub-classes give the
details.

If you want to wire, then you misunderstand the way how search with
facets etc. works. What you want is not a chaining of queries, but
instead a filtering of the set. This filtering is important as it
narrows the resultset where the search is applied onto.

In java-api one usually starts with getting an
SearchRequestBuilder,

then puts an (only one) query to it and then adds filters to it.

Simply said something like this:

private SearchRequestBuilder initBuilder(SearchRequestBuilder srq)
{

    QueryStringQueryBuilder query =

QueryBuilders.queryString(queryString).useDisMax(true).defaultOperator(operator);

    //boosting
    for (Map.Entry<String, Float> e : boostFields.entrySet()) {
        query.field(e.getKey(), e.getValue());
    }
    srq.setQuery(query); <- only this is the Query!!!
    srq.addFields(fields.toArray(new String[fields.size()]));
    //sorting
    for (Map.Entry<String, SortOrder> e : sortMap.entrySet()) {
        srq.addSort(e.getKey(), e.getValue());
    }
    //filtering <- this is what you want
    AndFilterBuilder myFilters = FilterBuilders.andFilter();
    if (termFilters.size() > 0) {
        for (Map.Entry<String, String> e :

termFilters.entrySet())

{
myFilters.add(FilterBuilders.termFilter(e.getKey(),
e.getValue()));
}
srq.setFilter(myFilters);
}

    for (String facet : termFacets.keySet()) {
        TermsFacetBuilder tf =

FacetBuilders.termsFacet(facet).field(facet).size(termFacets.get(facet)).order(getTermsComparator());

        if (termFilters.size() > 0) {
            tf.facetFilter(myFilters);
        }
        srq.addFacet(tf);
    }
    return srq;
}

please note that this is just an example from my main usage within
an

wicket-dataprovider, but it should you show how to tighten things.
Be

also sure to cleanse any data you pass to ES prior giving it to the
query:

/**
* function to escape the string from bad chars for the search
*
* @param str the String that should be escaped
* @return an escaped String
*/
@SuppressWarnings({"ConstantConditions"})
public static String smartEscapeQuery(String str) {
StringBuilder sb = new StringBuilder();
for (int i = 0; i < str.length(); i++) {
char c = str.charAt(i);
if (c == '\' || c == '+' || c == '-' || c == '!' || c
==

'(' || c == ')' || c == ':'
|| c == '^' || c == '[' || c == ']' || c ==
'"'

|| c == '{' || c == '}' || c == '~'
|| c == '?' || c == '|' || c == '&' || c == ';'
|| (!Character.isSpaceChar(c) &&
Character.isWhitespace(c))) {
sb.append('\');
}
sb.append(c);
}
return sb.toString();
}

Best,

KB

PS: don't even think of putting a own wrapper around ES<->Java as
you

will not even come close to its speed and stability (ES JavaApi
joins

the es node network, meaning full clustering, optimized data and
request traveling etc;)

On 21 Nov., 22:20, Ali loghm...@gmail.com wrote:

Hi mates,
I have been asked to use elasticsearch JAVA API (I personally
love

its

REST API, but I am not the decision maker here). I have been
struggling with finding a good set of classes/methods to build
comprehensive queries. I have already looked into
"BaseQueryBuilder"

and its implementing classes but none allow wiring of multiple
query

clauses!

I thought maybe the JAVA API is not there yet in terms of
milestones,

product management, resources .. (I believe what the team has
delivered so far is AWESOME, and JAVA API is really just a 2010-
last

decade thing). But, like I said I am asked to use JAVA API. So,
please

help me to find a solution for this. Either I have to develop a
JAVA

REST wrapper around the JSON REST API or find a way to use the
current

JAVA API to build comprehensive queries.

Any help is very much appreciated! :wink:
Thanks,
Ali


(system) #10