Same query, different CPU util when run with Java API versus REST

Jeff_Potts · December 12, 2014, 5:42pm

I have two JMeter tests. Let's call them "Straight Elasticsearch Test" and
"Service Test". Both tests use the same settings in terms of number of
threads (users), number of times the URL is invoked (loops), and randomized
CSV-based data set.

The difference between the two tests is that the Straight Elasticsearch
Test just makes HTTP requests against my cluster whereas the Service Test
makes HTTP requests against my Java service layer.

Both tests invoke the same Elasticsearch query. I've verified this by
comparing what the Straight Elasticsearch Test posts to what the Java
service creates using the SearchRequestBuilder. Here is the query and the
search invocation via the Java API:

gist.github.com

https://gist.github.com/jpotts/feaf0cedc2eea29a8aa7

SearchExample.java

public List<Content> fetchContentForPlacements(Date curDate, String relString, String pageId, Metadata meta) throws ContentServiceException {

	// check for required parameters
	if (pageId == null) {
		throw new MissingParameterException(Constants.MSG_PAGE_ID_REQUIRED);
	}

	// default the current date to now if it wasn't provided
	if (curDate == null) {
		curDate = new Date();

This file has been truncated. show original

query.json

{
  "size" : 200,
  "query" : {
    "function_score" : {
      "functions" : [ {
        "filter" : {
          "term" : {
            "loginState" : "all"
          }
        },

This file has been truncated. show original

I'm running both tests with 10 concurrent users and 2000 iterations for a
total of 20,000 hits against either Elasticsearch directly or the Java
service depending on the test.

Here's the puzzling thing:
The Straight Elasticsearch Test causes the CPU on my single-node cluster to
go to around 50% for the duration of the test. But the Service Test causes
it to go to twice that.

So the same number of concurrent users, the same number of total hits, and
the same query are causing drastically different CPU utilization rates.

What could be causing this?

I've tested with the Java service using the Transport Client as well as the
Node Client, the results are the same.

I'm thinking it must be something different in how the server handles HTTP
request versus Node/Transport Client requests.

This is Elasticsearch 1.3.4 and the index is a single shard with no
replicas and about 100k docs.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/63fd9d8e-ec67-4c6c-817f-4201d5f18ea9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jeff_Potts · December 12, 2014, 5:44pm

I should mention that the Elasticsearch node, the Java service, and the
JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · December 14, 2014, 5:52pm

Can you post the full query code for better recreation?

Jörg

On Fri, Dec 12, 2014 at 6:44 PM, Jeff Potts jeffpotts01@gmail.com wrote:

I should mention that the Elasticsearch node, the Java service, and the
JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGvO8iOu0%3DqfmNegSuB2Pv%3DD8y2pQuXem9HUj4%2BV6vO1g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Jeff_Potts · December 15, 2014, 2:27pm

Yes, updated the gist. Thanks for taking a look at this.

Jeff

On Sunday, December 14, 2014 11:52:35 AM UTC-6, Jörg Prante wrote:

Can you post the full query code for better recreation?

Jörg

On Fri, Dec 12, 2014 at 6:44 PM, Jeff Potts <jeffp...@gmail.com
<javascript:>> wrote:

I should mention that the Elasticsearch node, the Java service, and the
JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c312478a-e5c5-49aa-abd8-88eeac2c2051%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Marie_Jacob · December 16, 2014, 8:01pm

I have this same issue, except that CPU utilization is approximately the
same, but response times are very different. I'm using ES 1.3.4. I have two
JMeter tests simulating concurrent tests with the exact same configuration.
The test using the Java Sampler with the Transport Client are showing
response times that are twice as much as the test with HTTP samplers. Are
the connection channels in the ES client maybe overloaded?

Any insight on this would definitely be great.

Thanks,
Marie.

On Monday, December 15, 2014 9:27:18 AM UTC-5, Jeff Potts wrote:

Yes, updated the gist. Thanks for taking a look at this.

Jeff

On Sunday, December 14, 2014 11:52:35 AM UTC-6, Jörg Prante wrote:

Can you post the full query code for better recreation?

Jörg

On Fri, Dec 12, 2014 at 6:44 PM, Jeff Potts jeffp...@gmail.com wrote:

I should mention that the Elasticsearch node, the Java service, and the
JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7ff35b47-9e32-4f85-8899-2bbef4e632e8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

brian_yoder · December 16, 2014, 8:38pm

Jeff,

Does getReadClient() get a reference to a previously created singleton
TransportClient (or NodeClient, as the case may be)? I am guessing yes, but
just asking to be sure.

In my own set-up, I have created a thin HTTP REST layer (also using Netty,
with the LMAX Disruptor to minimize thread usage while maximizing
throughput). It contains our business logic, and issues queries and updates
to Elasticsearch via the TransportClient. It creates a singleton instance
of the TransportClient, and shares this singleton's reference throughout
the service. Using seige, I have hammered the combination of my thin server
plus Elasticsearch and don't see any performance issues as you describe.

Just a thought...

Regards,
Brian

On Monday, December 15, 2014 9:27:18 AM UTC-5, Jeff Potts wrote:

Yes, updated the gist. Thanks for taking a look at this.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e0f039df-95fc-4fb9-a787-84ee56d77e84%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jeff_Potts · December 16, 2014, 9:06pm

Yes, getReadClient() gets a Node Client that is instantiated by Spring and
then injected as a dependency. I have tried the Transport Client as well
and it makes no difference.

An interesting finding is that I have isolated the performance degradation
to the Range filter. When my service has these filters the test using REST
causes the ES node to use 9% CPU while the Java service uses 22% CPU:

filtersToApply.add(FilterBuilders.termFilter(Constants.PROP_PAGE_ID,
pageId));
filtersToApply.add(FilterBuilders.termFilter(Constants.PROP_GEO_CODE,
meta.getGeoCode()));
filtersToApply.add(FilterBuilders.termFilter(Constants.PROP_PLACEMENT_ID,
meta.getPlacementId()));

When I add a date range, the difference in performance increases
dramatically with the REST call staying at around 9% and the Java service
driving the CPU util to 81%:

filtersToApply.add(FilterBuilders.rangeFilter(Constants.PROP_PUB_DATE).lte(curDate.getTime()));

I've also noted that while the index search shard query rate is identical
between the two tests, the total shard search query time is dramatically
different with the Java service driving a much higher query time compared
to the REST-based test.

Jeff

On Tuesday, December 16, 2014 2:38:51 PM UTC-6, Brian wrote:

Jeff,

Does getReadClient() get a reference to a previously created singleton
TransportClient (or NodeClient, as the case may be)? I am guessing yes, but
just asking to be sure.

In my own set-up, I have created a thin HTTP REST layer (also using Netty,
with the LMAX Disruptor to minimize thread usage while maximizing
throughput). It contains our business logic, and issues queries and updates
to Elasticsearch via the TransportClient. It creates a singleton instance
of the TransportClient, and shares this singleton's reference throughout
the service. Using seige, I have hammered the combination of my thin server
plus Elasticsearch and don't see any performance issues as you describe.

Just a thought...

Regards,
Brian

On Monday, December 15, 2014 9:27:18 AM UTC-5, Jeff Potts wrote:

Yes, updated the gist. Thanks for taking a look at this.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/dd88c5a7-28ec-4aa2-ad1f-2974a5ed1978%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Jeff_Potts · December 16, 2014, 10:04pm

Scratch what I said about total shard search query time. I had started a
client node and was pointing my REST test at that.

CPU difference still there.

Also, I've upgraded to 1.3.7.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e429e400-2be3-4481-8218-c3a0e3d76308%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · December 16, 2014, 10:57pm

The difference between DSL and Java is that in Java you use search type
QUERY_AND_FETCH which is slow.

Jörg

On Mon, Dec 15, 2014 at 3:27 PM, Jeff Potts jeffpotts01@gmail.com wrote:

Yes, updated the gist. Thanks for taking a look at this.

Jeff

On Sunday, December 14, 2014 11:52:35 AM UTC-6, Jörg Prante wrote:

Can you post the full query code for better recreation?

Jörg

On Fri, Dec 12, 2014 at 6:44 PM, Jeff Potts jeffp...@gmail.com wrote:

I should mention that the Elasticsearch node, the Java service, and the
JMeter test client are all on different machines.

Jeff

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/20ee0ee8-5b4c-476e-8403-1db175de4158%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c312478a-e5c5-49aa-abd8-88eeac2c2051%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c312478a-e5c5-49aa-abd8-88eeac2c2051%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGE8DGWF-b1o9StQkY9dSFDDLvtyXGifAGtAx%2BC7Jx96Q%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Jeff_Potts · December 17, 2014, 12:27am

I figured it out. The queries executed by the straight Elasticsearch REST
test and the Java test are not exactly identical. The difference is that
the "current date" used by the straight Elasticsearch test is pulled from a
randomized CSV file whereas the Java service always injects the actual
current date down to the millisecond. So, even though I switched the query
and the Java to leverage filters, the straight Elasticsearch REST test is
the only test that takes advantage of the cache. The Java service query
doesn't take advantage of the cache because the current date changes every
millisecond.

To confirm this I hardcoded the milliseconds in both the REST test and the
Java service. Once I did that, the CPU rates (and everything else) were
identical.

The other telltale sign that there was a problem was the indices filter
cache graph in Marvel. Once straight REST test ran once, it could be re-run
and never cause an increase in what is in the filter cache. The Java
service, on the other hand, was populating the filter cache every time.

Thanks to everyone that weighed in on this.

Jeff

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4a3e2981-7a3b-4e6e-a2f0-ed65fddc9150%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.