Thanks for your reply.
See these graphs for 12 hours of G1 vs Non-G1
I must say i like the G1's results much better.
GC counts and time spent gc'ing are much better, and overal cpu load is
also a tad lower.
The heap usage pattern looks radically different.
Blue Line -> jre7u6 with these settings:
Bigdesk GC stats:
Uptime: 12 hours, 29min
Total time: 860'092ms
Total count: 20'425
Green Line -> jre7u6 with the G1 enabled:
Bigdesk GC stats:
Uptime: 12 hours, 28min
Total time: 80'736ms
Total count: 1'868
Regarding the SurviverRatio: I've removed it now. It was a leftover from a
test run with a much bigger machine (70GB of RAM) where I tried to limit
the sizes of the surviver spaces.
The thing that bites us right now is that the CPU load is ever increasing
until we restart the java process.
Having to restart ever 12~24 hours can't be normal procedure. Any ideas on
how to identify the cause of this increasing load would be much appreciated.
On Thursday, August 16, 2012 12:53:50 AM UTC+2, Otis Gospodnetic wrote:
I'd drop G1. In our tests it hasn't proven to work well. Regarding Java
7 - we have clients who are using ES with Java 7 without issues.
You are using -XX:SurvivorRatio=16 on purpose?
Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html
On Wednesday, August 15, 2012 9:02:09 AM UTC-4, Toni Birrer wrote:
I'm now running a elasticsearch cluster with 5 nodes on java 7 (
jre1.7.0_04 ) using the new G1 garbage collector.
We use the latest version of elasticsearch, 0.19.8
Here's an overview of our setup: http://www.screencast.com/t/wwW6Pcua
I use these java settings:
Java Additional Parameters
Initially the nodes run great. Avg. query time is ~ 12ms. But the cpu
load grows slowly, (without affecting the query time) but eventually
overwhelming the cpu at which time the cluster starts freaking out and the
response times grow into the seconds. As a very ugly workaround we simply
recycle the java processes every 12 hours, in a rolling fashion across the
entire cluster. After the restart the cpu usage drops very low for that
node, and then starts to creep up again.
We graph the whole thing with statsd/graphite, here's a bunch of graphs
that show the effect:
process cpu usage: http://www.screencast.com/t/dun2dpNJdPh
heap usage: http://www.screencast.com/t/7DeB2Mqp
gc count: http://www.screencast.com/t/evMK6zUPQm
gc time: http://www.screencast.com/t/9rwv9WxxBfKs
the actual work load doesn't seem related:
These are ec2 m1.xlarge (16gb ram, 4 cores)
What could cause this? Maybe I'm wrong, but I doesn't look to me like the
garbage collection is to blame here. Compare to some values I've seen with
Java 6 and without G1 these gc counts are very acceptable.
Is it even recommended to run elasticsearch on Java 7? Do you use it with
or without G1?
Are there any best-practice java7 settings someone is willing to share?