ElasticSeach full CPU usage

Elastic Stack Elasticsearch
1

Hello,

I'm currently using ElasticSearch 0.19.11 deployed into CentOS 5.7 (kernel
2.6.18-274.18.1.el5xen) boxes having 4GB or RAM/4 cores. The environment
contains 2 such boxes and ElasticSearch is not clustered (effectively the
two being independent). From time to time (once every approximately 3 or 4
days) one of the machines (apparently in a random order) heavily degrades
it's performance up to full CPU consumption.

The Java process running the ElasticSeach is configured with: "-Xms1g
-Xmx1g -Xss256k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly"
parameters.* *Each instance has a single index and contains ~1K documents.

Attached are some facts observed while experiencing the high CPU
consumption: jstat.txt, jstack.txt, top.txt, hotthreads.txt and the
elasticsearch.yml config we are using.

Can anyone help?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

The PID of the ES process was 32024.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

What JVM version do you use?

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hello Jörg,

Thanks for your reply, here is the version of the Java currently installed
("java -version"):

java version "1.6.0_26"

Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) 64-Bit Server VM (build 20.1-b02, mixed mode)

Do you have any suspects?

Regards,
Marius Span.

On Tuesday, September 24, 2013 1:19:37 PM UTC+3, Span Marius wrote:

Hello,

I'm currently using Elasticsearch 0.19.11 deployed into CentOS 5.7 (kernel
2.6.18-274.18.1.el5xen) boxes having 4GB or RAM/4 cores. The environment
contains 2 such boxes and Elasticsearch is not clustered (effectively the
two being independent). From time to time (once every approximately 3 or 4
days) one of the machines (apparently in a random order) heavily degrades
it's performance up to full CPU consumption.

The Java process running the ElasticSeach is configured with: "-Xms1g
-Xmx1g -Xss256k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly"
parameters.* *Each instance has a single index and contains ~1K documents.

Attached are some facts observed while experiencing the high CPU
consumption: jstat.txt, jstack.txt, top.txt, hotthreads.txt and the
elasticsearch.yml config we are using.

Can anyone help?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

I can't see what is happening exactly, but it looks like you ran into lock
contention at random which is not related to ES or Lucene but to the
underlying JVM. Can the issue be reproduced?

You should try if increasing the heap memory works because Java 6 has some
memory related issues.

Java 6u26 is quite old. It should be updated to the latest Java 7.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

Jörg,

Unfortunately I did not manage to create a deterministic reproduction path.
Currently this is happening once every 3 to 5 days on random boxes.

Thanks for your suggestion, we'll do this together with an ES upgrade as
soon as possible.
Until than, do you see any other thing being suspicious?

Marius.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Except that 0.19.11 is the version I also want to upgrade asap to 0.90+, I
can't see any peculiarities. 0.19 is a bit memory hungry when it uses
caches/facets.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.