Very high CPU usage of ES in idle state

Elastic Stack Elasticsearch
1

I have set up a 1 node cluster of ES using ES version 0.19.11 for storing
some logs. After inserting about 5000 logs I now see that my CPU usage is
rocketing above 100% constanltly even when no/very few search queries are
reaching it. ES has become very unresponsive and some of my search queries
randomly return the following exception.

"error" : "EsRejectedExecutionException[rejected execution of
[org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]",
"status" : 503

By randomly i mean sometimes they succeed but mostly they fail.

Here is my ES config:

index.cache.field.expire: 20m
index.cache.field.type: soft

bootstrap.mlockall: true
threadpool:
search:
type: fixed
size: 20
queue_size: 1000
index:
type: fixed
size: 16
bulk:
type: fixed
size: 5

I did some digging into the issue. The stats for node_stats, hot_threads
api and jstack can be found in this gist

I am also attaching the results of bigdesk. The cluster health is green but
looking at these results, it seems that ES is very low on heap memory. But
i am unable to understand the constant high CPU usage even in the idle
state. How can this issue be solved?

Kindly give some context and explanation to any answer.

Thanks
-atk

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Note, for your workload of around 100 qps, I have the impression you
have selected small memory for heap, around 350MB (?), and your GC is
around 10-20 seconds, but your RAM is total of 2 GB. This indicates you
should consider increasing the heap. GC times are going through the roof
and they give you the high CPU and unresponsiveness.

On a machine of 2 GB RAM there is a higher risk of bootstrap.mlockall:
true to starve your overall system processes because you will quickly
force them into swapping/paging, or you run out of real memory. So
consider removing this option to give the system enough room or upgrade
to a machine with > 4 GB RAM. Another option is adding more nodes of course.

If you stay with 2 GB RAM I would also recommend to disable 64-bit
server JVM with "java -d32 -client" in order to save some bits on the
heap. 64-bit server JVM makes sense on machines where you can use
virtual address space in the area of > 4GB RAM.

Jörg

Am 09.04.13 12:07, schrieb Talha Khan:

I have set up a 1 node cluster of ES using ES version 0.19.11 for
storing some logs. After inserting about 5000 logs I now see that my
CPU usage is rocketing above 100% constanltly even when no/very few
search queries are reaching it. ES has become very unresponsive and
some of my search queries randomly return the following exception.
"error" : "EsRejectedExecutionException[rejected execution of
[org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]",
"status" : 503

By randomly i mean sometimes they succeed but mostly they fail.

Here is my ES config:

index.cache.field.expire: 20m
index.cache.field.type: soft

bootstrap.mlockall: true
threadpool:
search:
type: fixed
size: 20
queue_size: 1000
index:
type: fixed
size: 16
bulk:
type: fixed
size: 5

I did some digging into the issue. The stats for node_stats,
hot_threads api and jstack can be found in this gist

Elasticsearch high cpu · GitHub

I am also attaching the results of bigdesk. The cluster health is
green but looking at these results, it seems that ES is very low on
heap memory. But i am unable to understand the constant high CPU usage
even in the idle state. How can this issue be solved?

Kindly give some context and explanation to any answer.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

I will try out the recommendations that you suggested. Will post my results.

On Tuesday, 9 April 2013 19:27:04 UTC+5, Jörg Prante wrote:

Note, for your workload of around 100 qps, I have the impression you
have selected small memory for heap, around 350MB (?), and your GC is
around 10-20 seconds, but your RAM is total of 2 GB. This indicates you
should consider increasing the heap. GC times are going through the roof
and they give you the high CPU and unresponsiveness.

On a machine of 2 GB RAM there is a higher risk of bootstrap.mlockall:
true to starve your overall system processes because you will quickly
force them into swapping/paging, or you run out of real memory. So
consider removing this option to give the system enough room or upgrade
to a machine with > 4 GB RAM. Another option is adding more nodes of
course.

If you stay with 2 GB RAM I would also recommend to disable 64-bit
server JVM with "java -d32 -client" in order to save some bits on the
heap. 64-bit server JVM makes sense on machines where you can use
virtual address space in the area of > 4GB RAM.

Jörg

Am 09.04.13 12:07, schrieb Talha Khan:

I have set up a 1 node cluster of ES using ES version 0.19.11 for
storing some logs. After inserting about 5000 logs I now see that my
CPU usage is rocketing above 100% constanltly even when no/very few
search queries are reaching it. ES has become very unresponsive and
some of my search queries randomly return the following exception.
"error" : "EsRejectedExecutionException[rejected execution of

[org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]",

"status" : 503

By randomly i mean sometimes they succeed but mostly they fail.

Here is my ES config:

index.cache.field.expire: 20m
index.cache.field.type: soft

bootstrap.mlockall: true
threadpool:
search:
type: fixed
size: 20
queue_size: 1000
index:
type: fixed
size: 16
bulk:
type: fixed
size: 5

I did some digging into the issue. The stats for node_stats,
hot_threads api and jstack can be found in this gist

Elasticsearch high cpu · GitHub

I am also attaching the results of bigdesk. The cluster health is
green but looking at these results, it seems that ES is very low on
heap memory. But i am unable to understand the constant high CPU usage
even in the idle state. How can this issue be solved?

Kindly give some context and explanation to any answer.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.