Very high CPU usage of ES in idle state

Talha_Khan · April 9, 2013, 10:07am

I have set up a 1 node cluster of ES using ES version 0.19.11 for storing
some logs. After inserting about 5000 logs I now see that my CPU usage is
rocketing above 100% constanltly even when no/very few search queries are
reaching it. ES has become very unresponsive and some of my search queries
randomly return the following exception.

"error" : "EsRejectedExecutionException[rejected execution of
[org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]",
"status" : 503

By randomly i mean sometimes they succeed but mostly they fail.

Here is my ES config:

index.cache.field.expire: 20m
index.cache.field.type: soft

bootstrap.mlockall: true
threadpool:
search:
type: fixed
size: 20
queue_size: 1000
index:
type: fixed
size: 16
bulk:
type: fixed
size: 5

I did some digging into the issue. The stats for node_stats, hot_threads
api and jstack can be found in this gist

gist.github.com

https://gist.github.com/ahmedtalhakhan/5343784

hot_threads

::: [Demo Rackspace Server][h5EUaG_6R1WfEaG87VLkGQ][inet[/50.56.234.172:9300]]{master=true}
   
    8.1% (40.3ms out of 500ms) cpu usage by thread 'elasticsearch[Demo Rackspace Server][management][T#16]'
     2/10 snapshots sharing following 14 elements
       org.elasticsearch.common.Strings.format1Decimals(Strings.java:1378)
       org.elasticsearch.common.unit.ByteSizeValue.toString(ByteSizeValue.java:133)
       org.elasticsearch.index.merge.MergeStats.toXContent(MergeStats.java:142)
       org.elasticsearch.action.admin.indices.status.IndicesStatusResponse.toXContent(IndicesStatusResponse.java:159)
       org.elasticsearch.rest.action.admin.indices.status.RestIndicesStatusAction$1.onResponse(RestIndicesStatusAction.java:78)
       org.elasticsearch.rest.action.admin.indices.status.RestIndicesStatusAction$1.onResponse(RestIndicesStatusAction.java:70)

This file has been truncated. show original

jstack

2013-04-09 07:21:20
Full thread dump Java HotSpot(TM) 64-Bit Server VM (23.1-b03 mixed mode):

"elasticsearch[Demo Rackspace Server][generic][T#549]" daemon prio=10 tid=0x00000000015f1800 nid=0x1c3f runnable [0x00007fc21bf7c000]
   java.lang.Thread.State: RUNNABLE
  at org.elasticsearch.common.collect.RegularImmutableMap.newLinkedEntry(RegularImmutableMap.java:113)
	at org.elasticsearch.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:65)
	at org.elasticsearch.common.collect.ImmutableMap.copyOf(ImmutableMap.java:293)
	at org.elasticsearch.common.settings.ImmutableSettings.<init>(ImmutableSettings.java:61)
	at org.elasticsearch.common.settings.ImmutableSettings.<init>(ImmutableSettings.java:54)

This file has been truncated. show original

node_stats

{
  "cluster_name" : "Logs",
  "nodes" : {
    "h5EUaG_6R1WfEaG87VLkGQ" : {
      "timestamp" : 1365494162627,
      "name" : "Demo Rackspace Server",
      "transport_address" : "",
      "hostname" : "Server",
      "attributes" : {
        "master" : "true"

This file has been truncated. show original

I am also attaching the results of bigdesk. The cluster health is green but
looking at these results, it seems that ES is very low on heap memory. But
i am unable to understand the constant high CPU usage even in the idle
state. How can this issue be solved?

Kindly give some context and explanation to any answer.

Thanks
-atk

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · April 9, 2013, 2:27pm

Note, for your workload of around 100 qps, I have the impression you
have selected small memory for heap, around 350MB (?), and your GC is
around 10-20 seconds, but your RAM is total of 2 GB. This indicates you
should consider increasing the heap. GC times are going through the roof
and they give you the high CPU and unresponsiveness.

On a machine of 2 GB RAM there is a higher risk of bootstrap.mlockall:
true to starve your overall system processes because you will quickly
force them into swapping/paging, or you run out of real memory. So
consider removing this option to give the system enough room or upgrade
to a machine with > 4 GB RAM. Another option is adding more nodes of course.

If you stay with 2 GB RAM I would also recommend to disable 64-bit
server JVM with "java -d32 -client" in order to save some bits on the
heap. 64-bit server JVM makes sense on machines where you can use
virtual address space in the area of > 4GB RAM.

Jörg

Am 09.04.13 12:07, schrieb Talha Khan:

I have set up a 1 node cluster of ES using ES version 0.19.11 for
storing some logs. After inserting about 5000 logs I now see that my
CPU usage is rocketing above 100% constanltly even when no/very few
search queries are reaching it. ES has become very unresponsive and
some of my search queries randomly return the following exception.
"error" : "EsRejectedExecutionException[rejected execution of
[org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]",
"status" : 503

By randomly i mean sometimes they succeed but mostly they fail.

Here is my ES config:

index.cache.field.expire: 20m
index.cache.field.type: soft

bootstrap.mlockall: true
threadpool:
search:
type: fixed
size: 20
queue_size: 1000
index:
type: fixed
size: 16
bulk:
type: fixed
size: 5

I did some digging into the issue. The stats for node_stats,
hot_threads api and jstack can be found in this gist

Elasticsearch high cpu · GitHub

I am also attaching the results of bigdesk. The cluster health is
green but looking at these results, it seems that ES is very low on
heap memory. But i am unable to understand the constant high CPU usage
even in the idle state. How can this issue be solved?

Kindly give some context and explanation to any answer.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Talha_Khan · April 9, 2013, 6:44pm

I will try out the recommendations that you suggested. Will post my results.

On Tuesday, 9 April 2013 19:27:04 UTC+5, Jörg Prante wrote:

Note, for your workload of around 100 qps, I have the impression you
have selected small memory for heap, around 350MB (?), and your GC is
around 10-20 seconds, but your RAM is total of 2 GB. This indicates you
should consider increasing the heap. GC times are going through the roof
and they give you the high CPU and unresponsiveness.

On a machine of 2 GB RAM there is a higher risk of bootstrap.mlockall:
true to starve your overall system processes because you will quickly
force them into swapping/paging, or you run out of real memory. So
consider removing this option to give the system enough room or upgrade
to a machine with > 4 GB RAM. Another option is adding more nodes of
course.

If you stay with 2 GB RAM I would also recommend to disable 64-bit
server JVM with "java -d32 -client" in order to save some bits on the
heap. 64-bit server JVM makes sense on machines where you can use
virtual address space in the area of > 4GB RAM.

Jörg

Am 09.04.13 12:07, schrieb Talha Khan:

I have set up a 1 node cluster of ES using ES version 0.19.11 for
storing some logs. After inserting about 5000 logs I now see that my
CPU usage is rocketing above 100% constanltly even when no/very few
search queries are reaching it. ES has become very unresponsive and
some of my search queries randomly return the following exception.
"error" : "EsRejectedExecutionException[rejected execution of

[org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]",

"status" : 503

By randomly i mean sometimes they succeed but mostly they fail.

Here is my ES config:

index.cache.field.expire: 20m
index.cache.field.type: soft

bootstrap.mlockall: true
threadpool:
search:
type: fixed
size: 20
queue_size: 1000
index:
type: fixed
size: 16
bulk:
type: fixed
size: 5

I did some digging into the issue. The stats for node_stats,
hot_threads api and jstack can be found in this gist

Elasticsearch high cpu · GitHub

I am also attaching the results of bigdesk. The cluster health is
green but looking at these results, it seems that ES is very low on
heap memory. But i am unable to understand the constant high CPU usage
even in the idle state. How can this issue be solved?

Kindly give some context and explanation to any answer.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Topic		Replies	Views
Elasticsearch high cpu usage Elasticsearch	2	370	July 6, 2017
All Primary Nodes Maxed out CPU Elasticsearch	3	424	June 9, 2020
Performance issue on some requests Elasticsearch	4	1654	April 10, 2017
Continuous high CPU and load + EsRejectedExecutionException Elasticsearch	2	488	November 8, 2018
High CPU usage / load average while no running queries Elasticsearch	16	23108	February 5, 2019

Very high CPU usage of ES in idle state

Related topics