Performance difference between REST and Java API

Marie_Jacob · December 18, 2014, 6:48pm

I'm benchmarking results for an ES cluster, using both the REST api and
native Java client. We're getting very different response times, between
each of these (the REST api is doing approximately 50% better in the 95th
percentile). I was wondering what the cause of this is, since the search
requests look identical to me (unless I missed something).

Here are the queries:

gist.github.com

https://gist.github.com/anonymous/509b3db873a30d8961ed

gistfile1.txt

curl -XPOST http://hostname:8080/index0/item/_search?routing=0_${catId}&_source=false
 
{"size": 20,"query": { 
  "filtered": { 
    "query": { "match": { "title": { 
        "query": "${title}", 
        "type": "boolean",
        "minimum_should_match": "60%"}}},
    "filter": {"bool": {"must": {"term": {"category": "${catId}"}
  }}}}}}

gistfile2.txt

BoolFilterBuilder filter = FilterBuilders.boolFilter().must(FilterBuilders.termFilter("category", cat));
 
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
                    .size(20)
                    .fetchSource(false)
                    .query(QueryBuilders.filteredQuery(
                            QueryBuilders.matchQuery("title", query)
                                    .minimumShouldMatch("60%"), filter));
 
SearchRequest srb = new SearchRequest()

This file has been truncated. show original

Our setup: 20 node cluster, 20 shards/2 replicas, and benchmarking from a
separate machine with JMeter (an HTTP sampler for each of the 20 nodes, and
a separate test plan with a Java sampler initializing the ES java client,
and sending queries).

Any help appreciated.

-M

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/a5915f2f-ed31-4abb-af68-77c67a586aa9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · December 18, 2014, 10:32pm

How about this?

gist.github.com

https://gist.github.com/anonymous/509b3db873a30d8961ed#comment-1359074

gistfile1.txt

curl -XPOST http://hostname:8080/index0/item/_search?routing=0_${catId}&_source=false
 
{"size": 20,"query": { 
  "filtered": { 
    "query": { "match": { "title": { 
        "query": "${title}", 
        "type": "boolean",
        "minimum_should_match": "60%"}}},
    "filter": {"bool": {"must": {"term": {"category": "${catId}"}
  }}}}}}

gistfile2.txt

BoolFilterBuilder filter = FilterBuilders.boolFilter().must(FilterBuilders.termFilter("category", cat));
 
SearchSourceBuilder sourceBuilder = new SearchSourceBuilder()
                    .size(20)
                    .fetchSource(false)
                    .query(QueryBuilders.filteredQuery(
                            QueryBuilders.matchQuery("title", query)
                                    .minimumShouldMatch("60%"), filter));
 
SearchRequest srb = new SearchRequest()

This file has been truncated. show original

Jörg

On Thu, Dec 18, 2014 at 7:48 PM, Marie Jacob jacob.marie@gmail.com wrote:

I'm benchmarking results for an ES cluster, using both the REST api and
native Java client. We're getting very different response times, between
each of these (the REST api is doing approximately 50% better in the 95th
percentile). I was wondering what the cause of this is, since the search
requests look identical to me (unless I missed something).

Here are the queries:

gist:509b3db873a30d8961ed · GitHub

Our setup: 20 node cluster, 20 shards/2 replicas, and benchmarking from a
separate machine with JMeter (an HTTP sampler for each of the 20 nodes, and
a separate test plan with a Java sampler initializing the ES java client,
and sending queries).

Any help appreciated.

-M

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a5915f2f-ed31-4abb-af68-77c67a586aa9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a5915f2f-ed31-4abb-af68-77c67a586aa9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFm1Eo-h6y%3D2NjvZtY7yGa3YRBcLZG%3DjAxAS9oxCKZeyw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Marie_Jacob · December 19, 2014, 12:18am

Thanks Jörg.

I tried that, and it seems to make no difference. All I could gather are
these graphs (attached) from the bigdesk plugin that are showing
performance for the query and fetch phase (the first is during ES java
client use, the second is using the REST calls) . It's really weird that
the FETCH phase is taking any time at all, if I set _source = false.

On Thursday, December 18, 2014 5:33:05 PM UTC-5, Jörg Prante wrote:

How about this?

gist:509b3db873a30d8961ed · GitHub

Jörg

On Thu, Dec 18, 2014 at 7:48 PM, Marie Jacob <jacob...@gmail.com
<javascript:>> wrote:

I'm benchmarking results for an ES cluster, using both the REST api and
native Java client. We're getting very different response times, between
each of these (the REST api is doing approximately 50% better in the 95th
percentile). I was wondering what the cause of this is, since the search
requests look identical to me (unless I missed something).

Here are the queries:

gist:509b3db873a30d8961ed · GitHub

Our setup: 20 node cluster, 20 shards/2 replicas, and benchmarking from a
separate machine with JMeter (an HTTP sampler for each of the 20 nodes, and
a separate test plan with a Java sampler initializing the ES java client,
and sending queries).

Any help appreciated.

-M

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/a5915f2f-ed31-4abb-af68-77c67a586aa9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a5915f2f-ed31-4abb-af68-77c67a586aa9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/060e3e34-1326-4d6d-a645-670924818d6c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · December 19, 2014, 8:20am

The idea is for tracing the issue to rebuild the query from simple to
complex and compare.

I would start with the query and add the filters step by step, to identify
the part which causes trouble.

On possible cause is that REST API is using some defaults which have to be
set in the Java API explicitly.

Jörg

On Fri, Dec 19, 2014 at 1:18 AM, Marie Jacob jacob.marie@gmail.com wrote:

Thanks Jörg.

I tried that, and it seems to make no difference. All I could gather are
these graphs (attached) from the bigdesk plugin that are showing
performance for the query and fetch phase (the first is during ES java
client use, the second is using the REST calls) . It's really weird that
the FETCH phase is taking any time at all, if I set _source = false.

On Thursday, December 18, 2014 5:33:05 PM UTC-5, Jörg Prante wrote:

How about this?

gist:509b3db873a30d8961ed · GitHub

Jörg

On Thu, Dec 18, 2014 at 7:48 PM, Marie Jacob jacob...@gmail.com wrote:

I'm benchmarking results for an ES cluster, using both the REST api and
native Java client. We're getting very different response times, between
each of these (the REST api is doing approximately 50% better in the 95th
percentile). I was wondering what the cause of this is, since the search
requests look identical to me (unless I missed something).

Here are the queries:

gist:509b3db873a30d8961ed · GitHub

Our setup: 20 node cluster, 20 shards/2 replicas, and benchmarking from
a separate machine with JMeter (an HTTP sampler for each of the 20 nodes,
and a separate test plan with a Java sampler initializing the ES java
client, and sending queries).

Any help appreciated.

-M

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/a5915f2f-ed31-4abb-af68-77c67a586aa9%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/a5915f2f-ed31-4abb-af68-77c67a586aa9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/060e3e34-1326-4d6d-a645-670924818d6c%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/060e3e34-1326-4d6d-a645-670924818d6c%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHRJSL_wJoj-Q%2BypXC3Ej79fNPC-6j-BEJ-LMpPXU8yDQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.