Es2.3.3 OOM,someone help

Elastic Stack Elasticsearch
1

this is my es jvm config: -Xms30g -Xmx30g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC
my machine memery is 128G.
oom log:
[2016-07-28 04:52:50,597][WARN ][monitor.jvm ] [data-10.175.86.133] [gc][old][3042504][2383] duration [4.5m], collections [2]/[4.5m], total [4.5m]/[55.5m], memory [25gb]->[24.4gb]/[29.8gb], all_pools {[young] [331.5mb]->[7.4
mb]/[1.1gb]}{[survivor] [143.5mb]->[0b]/[149.7mb]}{[old] [24.6gb]->[24.4gb]/[28.5gb]}
[2016-07-28 05:00:56,676][ERROR][transport.netty ] [data-10.175.86.133] failed to handle exception response [org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPinger$1@557f84bf]
java.lang.OutOfMemoryError: Compressed class space
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:760)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPinger$1.handleException(MasterFaultDetection.java:260)
at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:212)
at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:202)
at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
[2016-07-28 05:13:03,490][WARN ][monitor.jvm ] [data-10.175.86.133] [gc][old][3042505][2393] duration [20m], collections [10]/[24.3m], total [20m]/[1.2h], memory [24.4gb]->[24.9gb]/[29.8gb], all_pools {[young] [7.4mb]->[438.
4mb]/[1.1gb]}{[survivor] [0b]->[96.6mb]/[149.7mb]}{[old] [24.4gb]->[24.4gb]/[28.5gb]}

2

my es config:
index.cache.field.max_size: 50000
index.cache.field.expire: 6h
index.cache.field.type: soft

indices.fielddata.cache.size: 30%
indices.fielddata.cache.expire: 6h
indices.breaker.total.limit: 70%
indices.breaker.fielddata.limit: 50%
indices.breaker.request.limit: 5%

3

How much data, how many nodes.

4

44 indexs,"number_of_shards": "5, "number_of_replicas": "2"
total 679m docs, total store.size 2.5T
3 data nodes, 128G mem, 6 disks no raid, 24 core

5

jmap -histo
num #instances #bytes class name

1: 4712107 3395754152 [B
2: 60496288 3387792128 org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry
3: 46912094 1354359296 [Ljava.lang.Object;
4: 27819468 1335334464 java.lang.invoke.MethodHandleImpl$CountingWrapper
5: 43807384 1051377216 org.codehaus.groovy.util.FastArray
6: 1011941 803611184 [J
7: 24929593 797746976 java.util.HashMap$Node
8: 4867729 700953872 [Lorg.codehaus.groovy.util.ComplexKeyHashMap$Entry;
9: 21731223 695399136 java.lang.invoke.BoundMethodHandle$Species_LL
10: 14958132 598325280 java.lang.invoke.BoundMethodHandle$Species_L3
11: 4163854 429125768 [C
12: 11474280 367176960 org.codehaus.groovy.util.SingleKeyHashMap$Entry
13: 347830 361635424 [Lorg.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry;
14: 553901 336111768 [I
15: 5044587 322853568 java.lang.invoke.BoundMethodHandle$Species_L4IL4
16: 1781466 295495928 [Ljava.util.HashMap$Node;
17: 5041401 241987248 java.lang.invoke.BoundMethodHandle$Species_L5
18: 2473877 217701176 java.lang.reflect.Method
19: 5819669 186229408 java.lang.invoke.BoundMethodHandle$Species_L
20: 3668978 176110944 java.util.HashMap
21: 3242 173983752 [Ljava.util.concurrent.ConcurrentHashMap$Node;
22: 5215892 166908544 groovy.lang.MetaBeanProperty
23: 1749764 139981120 java.lang.reflect.Constructor
24: 3938142 126020544 java.lang.ref.WeakReference
25: 3130036 125201440 java.lang.invoke.BoundMethodHandle$Species_L4
26: 101235 120651144 [S
27: 2434113 116837424 org.codehaus.groovy.util.ReferenceType$SoftRef
28: 1391360 100177920 java.lang.reflect.Field
29: 4172368 100136832 org.codehaus.groovy.util.SingleKeyHashMap
30: 658757 94861008 org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame

is groovy cause?

6

hot_thread:
11.8% (58.9ms out of 500ms) cpu usage by thread 'elasticsearch[data-*][index][T#14]'
3/10 snapshots sharing following 24 elements
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:265)
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254)
org.elasticsearch.script.groovy.GroovyScriptEngineService$3.run(GroovyScriptEngineService.java:191)
org.elasticsearch.script.groovy.GroovyScriptEngineService$3.run(GroovyScriptEngineService.java:185)
java.security.AccessController.doPrivileged(Native Method)
org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:185)
org.elasticsearch.script.ScriptService.compileInternal(ScriptService.java:311)
org.elasticsearch.script.ScriptService.compile(ScriptService.java:257)
org.elasticsearch.script.ScriptService.executable(ScriptService.java:442)
org.elasticsearch.action.update.UpdateHelper.executeScript(UpdateHelper.java:250)
org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:197)
org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:80)
org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:174)
org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:168)
org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:66)
org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(TransportInstanceSingleOperationAction.java:244)
org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(TransportInstanceSingleOperationAction.java:240)
org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33)
org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75)
org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300)
org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)

7

solution: