CPU Usage of elasticsearch process is almost 400% while doing performance Testing

Elastic Stack Elasticsearch
1

Hi Team,

I have following Setup of elastic search:
ES Version: elasticsearch-0.90.7
I have one Index having 20k crawled web links.

Machine Details:
OS Version: Linux 2.6.32-300.4.1.el5uek
Hardware: 6 x Intel(R) Xeon(R) CPU X5660 @ 2.80GHz/23GB

I am using Jmeter script to execute the simple search query on ES. I am
using 500 threads that will ramp up in 50 second and it will run in loop of
100..
So around 50,000 queries are being fired on ES in 120 second. i.e. Around
400 queries per second.
Problem is in this Testing the CPU usage of ES is around 400% and the
response time that we are getting from ES is 400+ms and it is also coming
as 1k ms and 2k ms..

Questions:

  1. Could you please tell me what might be going wrong here?
  2. What configuration I need to do at ES to achieve such kind of Testing?
  3. What type of system configuration I need to achieve this kind of testing?

Any other information you need then please let us know.

Regards,
Jayesh Bhoyar

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/989674c4-6e0a-433e-9b4f-752c0d7da13b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

What about your system memory and heap size stats?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 11 December 2013 14:21, Jayesh Bhoyar jsbonline2006@gmail.com wrote:

Hi Team,

I have following Setup of Elasticsearch:
ES Version: elasticsearch-0.90.7
I have one Index having 20k crawled web links.

Machine Details:
OS Version: Linux 2.6.32-300.4.1.el5uek
Hardware: 6 x Intel(R) Xeon(R) CPU X5660 @ 2.80GHz/23GB

I am using Jmeter script to execute the simple search query on ES. I am
using 500 threads that will ramp up in 50 second and it will run in loop of
100..
So around 50,000 queries are being fired on ES in 120 second. i.e. Around
400 queries per second.
Problem is in this Testing the CPU usage of ES is around 400% and the
response time that we are getting from ES is 400+ms and it is also coming
as 1k ms and 2k ms..

Questions:

  1. Could you please tell me what might be going wrong here?
  2. What configuration I need to do at ES to achieve such kind of Testing?
  3. What type of system configuration I need to achieve this kind of
    testing?

Any other information you need then please let us know.

Regards,
Jayesh Bhoyar

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/989674c4-6e0a-433e-9b4f-752c0d7da13b%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624b_cmCL1O1A2Tz%2BU9w-SS7xHVkT1y0VW8k4k6Wpe3svyw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

System Memory is: 23 GB.
How to find Heap Size Stats?

On Wednesday, December 11, 2013 8:53:59 AM UTC+5:30, Mark Walkom wrote:

What about your system memory and heap size stats?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fc2d6db5-0bc9-43f3-927b-e673482fa92b%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

You might want to install a monitor plugin, something like elastichq, kopf
or bigdesk.

These all make use of the various stats APIs and turn them into presentable
dashboards.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 11 December 2013 14:41, Jayesh Bhoyar jsbonline2006@gmail.com wrote:

System Memory is: 23 GB.
How to find Heap Size Stats?

On Wednesday, December 11, 2013 8:53:59 AM UTC+5:30, Mark Walkom wrote:

What about your system memory and heap size stats?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/fc2d6db5-0bc9-43f3-927b-e673482fa92b%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bR79kq7ba2mTt7NeKk5%3DWv__CfD4_Gc6R43dNWhDzutQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hi Mark,

Please find attached the complete screen shot of BigDesk when I am running
the Jmeter Testing

Regards,
Jayesh Bhoyar

On Wednesday, December 11, 2013 9:13:32 AM UTC+5:30, Mark Walkom wrote:

You might want to install a monitor plugin, something like elastichq, kopf
or bigdesk.

These all make use of the various stats APIs and turn them into
presentable dashboards.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0c2d23ba-bdb6-4f97-9f9a-734952b36bde%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Why do you think there is a problem? Seems ok to me.
What were you expecting and why?

On Wed, Dec 11, 2013 at 2:36 AM, Jayesh Bhoyar jsbonline2006@gmail.comwrote:

Hi Mark,

Please find attached the complete screen shot of BigDesk when I am running
the Jmeter Testing

Regards,
Jayesh Bhoyar

On Wednesday, December 11, 2013 9:13:32 AM UTC+5:30, Mark Walkom wrote:

You might want to install a monitor plugin, something like elastichq,
kopf or bigdesk.

These all make use of the various stats APIs and turn them into
presentable dashboards.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0c2d23ba-bdb6-4f97-9f9a-734952b36bde%40googlegroups.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFjHw37TJoa%3Dx-dOQ%3DQCnKp%3DPKaM%2BhcWwxmUcXyCkQdY2SDf7Q%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Hi,

Actually the response time I am getting for such load is 1000-2000 ms. And
Average is more than 600ms.

I want to know what configurations I should make on ES so that It will give
lesser response time such as 100-300 ms on peak load.

My peak load will be 500 queries per sec.

Regards,
Jayesh Bhoyar
+91-9011083120

On Thu, Dec 12, 2013 at 6:56 AM, Randall McRee randall.mcree@gmail.comwrote:

Why do you think there is a problem? Seems ok to me.
What were you expecting and why?

On Wed, Dec 11, 2013 at 2:36 AM, Jayesh Bhoyar jsbonline2006@gmail.comwrote:

Hi Mark,

Please find attached the complete screen shot of BigDesk when I am
running the Jmeter Testing

Regards,
Jayesh Bhoyar

On Wednesday, December 11, 2013 9:13:32 AM UTC+5:30, Mark Walkom wrote:

You might want to install a monitor plugin, something like elastichq,
kopf or bigdesk.

These all make use of the various stats APIs and turn them into
presentable dashboards.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPHRkZaHXUshUYKP5EXvzSMhnyQciPQV3MLKF92xyANZvd5AUg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

Hi,

This is a tinny index. You should be getting responses in single of maybe
double ms digits.
It's hard to give precise advice without details. Maybe your bottleneck is
the network? How is disk IO looking? Does speed improve over time? What
about GC? Is latency smaller with fewer concurrent queries? Can you set
refresh interval to, say, 60 seconds? What are your queries like? How
many docs do you return?

Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wednesday, December 11, 2013 11:12:39 PM UTC-5, Jayesh Bhoyar wrote:

Hi,

Actually the response time I am getting for such load is 1000-2000 ms. And
Average is more than 600ms.

I want to know what configurations I should make on ES so that It will
give lesser response time such as 100-300 ms on peak load.

My peak load will be 500 queries per sec.

Regards,
Jayesh Bhoyar
+91-9011083120

On Thu, Dec 12, 2013 at 6:56 AM, Randall McRee <randal...@gmail.com<javascript:>

wrote:

Why do you think there is a problem? Seems ok to me.
What were you expecting and why?

On Wed, Dec 11, 2013 at 2:36 AM, Jayesh Bhoyar <jsbonl...@gmail.com<javascript:>

wrote:

Hi Mark,

Please find attached the complete screen shot of BigDesk when I am
running the Jmeter Testing

Regards,
Jayesh Bhoyar

On Wednesday, December 11, 2013 9:13:32 AM UTC+5:30, Mark Walkom wrote:

You might want to install a monitor plugin, something like elastichq,
kopf or bigdesk.

These all make use of the various stats APIs and turn them into
presentable dashboards.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/66bce214-66ac-4d9e-b490-01086f424f8d%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

9

Please find attached the dashboard during this Jmeter Testing.

My First Email:

Hi Team,

I have following Setup of Elasticsearch:
ES Version: elasticsearch-0.90.7
I have one Index having 20k crawled web links.

Machine Details:
OS Version: Linux 2.6.32-300.4.1.el5uek
Hardware: 6 x Intel(R) Xeon(R) CPU X5660 @ 2.80GHz/23GB

I am using Jmeter script to execute the simple search query on ES. I am
using 500 threads that will ramp up in 50 second and it will run in loop of
100..
So around 50,000 queries are being fired on ES in 120 second. i.e. Around
400 queries per second.
Problem is in this Testing the CPU usage of ES is around 400% and the
response time that we are getting from ES is 400+ms and it is also coming
as 1k ms and 2k ms..

Questions:

  1. Could you please tell me what might be going wrong here?
  2. What configuration I need to do at ES to achieve such kind of Testing?
  3. What type of system configuration I need to achieve this kind of testing?

Any other information you need then please let us know.

Regards,
Jayesh Bhoyar

On Thu, Dec 12, 2013 at 10:34 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

This is a tinny index. You should be getting responses in single of maybe
double ms digits.
It's hard to give precise advice without details. Maybe your bottleneck
is the network? How is disk IO looking? Does speed improve over time?
What about GC? Is latency smaller with fewer concurrent queries? Can you
set refresh interval to, say, 60 seconds? What are your queries like? How
many docs do you return?

Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPHRkZYVazBJRpj-Pg0HMBWqFM2h%3DiLjJvqBfY6HvDQhnG4Grw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

10

Why not adding more nodes and more replicas?
I mean that if your queries use all your CPU ressources, you probably add more CPU to your cluster?

Check your GC as well. May be you are running GC too often which is using CPU and pause the JVM for a while.
Perhaps you need more memory here?

My 0.05 cents

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 12 déc. 2013 à 05:12, Jayesh Bhoyar jsbonline2006@gmail.com a écrit :

Hi,

Actually the response time I am getting for such load is 1000-2000 ms. And Average is more than 600ms.

I want to know what configurations I should make on ES so that It will give lesser response time such as 100-300 ms on peak load.

My peak load will be 500 queries per sec.

Regards,
Jayesh Bhoyar
+91-9011083120

On Thu, Dec 12, 2013 at 6:56 AM, Randall McRee randall.mcree@gmail.com wrote:

Why do you think there is a problem? Seems ok to me.
What were you expecting and why?

On Wed, Dec 11, 2013 at 2:36 AM, Jayesh Bhoyar jsbonline2006@gmail.com wrote:

Hi Mark,

Please find attached the complete screen shot of BigDesk when I am running the Jmeter Testing

Regards,
Jayesh Bhoyar

On Wednesday, December 11, 2013 9:13:32 AM UTC+5:30, Mark Walkom wrote:

You might want to install a monitor plugin, something like elastichq, kopf or bigdesk.

These all make use of the various stats APIs and turn them into presentable dashboards.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPHRkZaHXUshUYKP5EXvzSMhnyQciPQV3MLKF92xyANZvd5AUg%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4D1C15F3-F156-4C74-8ACA-D5BC1E1DEE5E%40pilato.fr.
For more options, visit https://groups.google.com/groups/opt_out.

1 Like