Search node getting slow down when Codecache is Full

[haandol]

my server, r3.2xlarge 8core 60G, is on the AWS. and I use obviously Elasticsearch 1.4 on JVM 1.8 with CMS GC for FGC and ParallelNewGC for YGC.

I run ES with these opts,

./bin/elasticsearch -Xms30g -Xmx30g -XX:NewRatio=2 -XX:ReservedCodeCacheSize=2g -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseParNewGC

just after run the ES, HQ said everything's fine and no slow query logs appear for a while.
but when about 8 hours past 'Codecache is Full' warning appears and ES search query is getting slow down horribly.
(approximately 200ms to 2s)

I just want to know why my codecache is full so often? and what should I do to avoid fulfilling codecache?

[nik9000 (Nik Everett)]

@haandol

I just want to know why my codecache is full so often? and what should I do to avoid fulfilling codecache?

Its really hard to tell from the what you've written but my first guess is dynamic scripting where you build parameters into the script so it has to be recompiled.

What do the code cache warning look like? What do your queries look like?

@haandol

my server, r3.2xlarge 8core 60G, is on the AWS. and I use obviously Elasticsearch 1.4 on JVM 1.8 with CMS GC for FGC and ParallelNewGC for YGC.

1.4 isn't an obvious version - firstly its not a complete version number, and secondly its reasonably old. Also, why are you using non-default GC settings? Did you run some analysis and find them better? Also I've never tried setting get settings that way. I imagine you've verified that they made their way to the java process but if you haven't its probably worth doing.