Hi,
Thanks a lot for the great replies, a lot for me to work on.
Some replies:
- Have you sized your ES cluster? How did you conclude that 8 nodes
are enough? Do you have configured caches/filter memory?
I was not part of this decision, to be honest I happened to have to
touch elasticsearch for the first time when the result of a long GC
and some other collateral damage made the cluster red. Since then I am
trying to understand what we can do better. I have to check about
caches/filter
We are going to experiment one node on openJDK 7
More difficult to do but we will at some point soon
- if you still get issues, try to reduce the "stop the world" duration
by carefully adapting the heap size to your workload, or by lowering
the memory pressure. You can scale ES over more nodes to achieve this
This is probably where I will put most of my efforts, more smaller
nodes, make shards max size not larger than heap size (hint from
another post, right now shard size is ~ 2.2 times the heap size), run
separate dedicated query, index, data and master nodes
- Can you share gc log files?
Unfortunately I don't have any (will enable), but I do have data in
graphite, perhaps if you can give me an idea of what you suspect you'd
find in the logs I could look for it in the graphs.
We don't pass this parameter so we must be running the defaults, the logs show:
[gc][ParNew][1686997][5685847] duration [6s], collections [6]/[13.5s],
total [6s]/[4.9d], memory [29.2gb]->[15.1gb]/[29.9gb], all_pools
{[Code Cache] [13.4mb]->[13.4mb]/[48mb]}{[Par Eden Space]
[125.6mb]->[64mb]/[216.3mb]}{[Par Survivor Space]
[27mb]->[27mb]/[27mb]}{[CMS Old Gen] [29.1gb]->[15gb]/[29.7gb]}{[CMS
Perm Gen] [46.1mb]->[46.1mb]/[82mb]}
- Enable gc logging on the server and run it through a tool like
gcviewer to help understand the heap behaviour, and something like
yourkit to see what's getting allocated (eg you might want to check
the filter cache to see if it's generating garbage due to a large
working set, or if lucene is doing a lot of merging).
Will do once we enable gc logging, thanks for the other suggestions
- do you use mmapfs and boostrap.mlockall
boostrap.mlockall
- These flags tell the JVM to begin a CMS collection when and only
when the old generation becomes more than 75% full. If your
application is allocating so rapidly that the old generation is filled
before the CMS collection as completed, then a "concurrent mode
failure" has occurred, and the JVM will fall back to a full
stop-the-world collection
I can see how this could be what's happening, we do indeed allocate
memory very rapidly, but I will go back and check the data before
jumping to conclusions, thanks a lot
Many thanks,
Simone
On Mon, Aug 12, 2013 at 10:59 PM, Jon Shea jonshea@foursquare.com wrote:
Hi Simone,
We had issues with long GC pauses too, but we were able to resolve them by
adjusting our JVM command line flags. Unfortunately, when I was
experimenting with solutions I wasn’t very disciplined about adjusting only
one variable at a time, so I can’t say for certain exactly what change
resolved our issue. But I have a good guess.
I think the thing that worked for us was removing the flags
-XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly.
These flags are included in ./bin/elasticsearch.in.sh that the
Elasticsearch launch script uses by default, but I strongly suspect they
should be removed. These flags tell the JVM to begin a CMS collection when
and only when the old generation becomes more than 75% full. If your
application is allocating so rapidly that the old generation is filled
before the CMS collection as completed, then a "concurrent mode failure" has
occurred, and the JVM will fall back to a full stop-the-world collection
which may have a long pause time.
I believe (but am not certain) that if you do not set these flags, then the
JVM will tune the CMSInitiatingOccupancyFraction on its own at runtime in an
effort to optimize the GC performance and avoid "concurrent mode failure". I
also think (but am not certain) that if you do set these flags, then the JVM
will assume you know better than it does, and will not tune them further at
runtime.
I also recommend you turn on GC logging. You can do this by including these
flags: -verbose:gc -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -Xloggc:<PATH TO LOG FILE>. The log format is not intuitive, but with enough Google searches you
can begin to make sense of it. If you’re using CMS and you see Full GC in
the GC logs move than once every few days, then something is amiss. My
current GC log goes back over a month, and I have only one Full GC in it.
Good luck,
-Jon
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.