# Es2.3.3 OOM,someone help **URL:** \ **Category:** Elasticsearch\ **Created:** [July 28, 2016, 1:30am UTC](https://discuss.elastic.co/t/es2-3-3-oom-someone-help/56574 "2016-07-28T01:30:04Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![win1027](https://avatars.discourse-cdn.com/v4/letter/w/7bcc69/32.png) [@win1027](https://discuss.elastic.co/u/win1027)\ **Post date:** [July 28, 2016, 1:30am UTC](https://discuss.elastic.co/t/es2-3-3-oom-someone-help/56574/1 "2016-07-28T01:30:04Z")
this is my es jvm config: -Xms30g -Xmx30g -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+HeapDumpOnOutOfMemoryError -XX:+DisableExplicitGC my machine memery is 128G. oom log: [2016-07-28 04:52:50,597][WARN][monitor.jvm] [data-10.175.86.133] [gc][old][3042504][2383] duration [4.5m], collections [2]/[4.5m], total [4.5m]/[55.5m], memory [25gb]-\>[24.4gb]/[29.8gb], all\_pools {[young] [331.5mb]-\>[7.4 mb]/[1.1gb]}{[survivor] [143.5mb]-\>[0b]/[149.7mb]}{[old] [24.6gb]-\>[24.4gb]/[28.5gb]} [2016-07-28 05:00:56,676][ERROR][transport.netty] [data-10.175.86.133] failed to handle exception response [org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPinger$1@557f84bf] java.lang.OutOfMemoryError: Compressed class space at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:760) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.elasticsearch.discovery.zen.fd.MasterFaultDetection$MasterPinger$1.handleException(MasterFaultDetection.java:260) at org.elasticsearch.transport.netty.MessageChannelHandler.handleException(MessageChannelHandler.java:212) at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:202) at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:136) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296) at org.jboss.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443) at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303) at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564) at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268) at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255) at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108) at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89) [2016-07-28 05:13:03,490][WARN][monitor.jvm] [data-10.175.86.133] [gc][old][3042505][2393] duration [20m], collections [10]/[24.3m], total [20m]/[1.2h], memory [24.4gb]-\>[24.9gb]/[29.8gb], all\_pools {[young] [7.4mb]-\>[438. 4mb]/[1.1gb]}{[survivor] [0b]-\>[96.6mb]/[149.7mb]}{[old] [24.4gb]-\>[24.4gb]/[28.5gb]} ---
**Author:** ![win1027](https://avatars.discourse-cdn.com/v4/letter/w/7bcc69/32.png) [@win1027](https://discuss.elastic.co/u/win1027)\ **Post date:** [July 28, 2016, 1:31am UTC](https://discuss.elastic.co/t/es2-3-3-oom-someone-help/56574/2 "2016-07-28T01:31:55Z")
my es config: index.cache.field.max\_size: 50000 index.cache.field.expire: 6h index.cache.field.type: soft indices.fielddata.cache.size: 30% indices.fielddata.cache.expire: 6h indices.breaker.total.limit: 70% indices.breaker.fielddata.limit: 50% indices.breaker.request.limit: 5% ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 28, 2016, 2:38am UTC](https://discuss.elastic.co/t/es2-3-3-oom-someone-help/56574/3 "2016-07-28T02:38:07Z")
How much data, how many nodes. ---
**Author:** ![win1027](https://avatars.discourse-cdn.com/v4/letter/w/7bcc69/32.png) [@win1027](https://discuss.elastic.co/u/win1027)\ **Post date:** [July 28, 2016, 3:45am UTC](https://discuss.elastic.co/t/es2-3-3-oom-someone-help/56574/4 "2016-07-28T03:45:25Z")
44 indexs,"number\_of\_shards": "5, "number\_of\_replicas": "2" total 679m docs, total store.size 2.5T 3 data nodes, 128G mem, 6 disks no raid, 24 core ---
**Author:** ![win1027](https://avatars.discourse-cdn.com/v4/letter/w/7bcc69/32.png) [@win1027](https://discuss.elastic.co/u/win1027)\ **Post date:** [July 28, 2016, 1:09pm UTC](https://discuss.elastic.co/t/es2-3-3-oom-someone-help/56574/5 "2016-07-28T13:09:51Z")
## jmap -histo num #instances#bytes class name 1: 4712107 3395754152 [B 2: 60496288 3387792128 org.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry 3: 46912094 1354359296 [Ljava.lang.Object; 4: 27819468 1335334464 java.lang.invoke.MethodHandleImpl$CountingWrapper 5: 43807384 1051377216 org.codehaus.groovy.util.FastArray 6: 1011941 803611184 [J 7: 24929593 797746976 java.util.HashMap$Node 8: 4867729 700953872 [Lorg.codehaus.groovy.util.ComplexKeyHashMap$Entry; 9: 21731223 695399136 java.lang.invoke.BoundMethodHandle$Species\_LL 10: 14958132 598325280 java.lang.invoke.BoundMethodHandle$Species\_L3 11: 4163854 429125768 [C 12: 11474280 367176960 org.codehaus.groovy.util.SingleKeyHashMap$Entry 13: 347830 361635424 [Lorg.codehaus.groovy.runtime.metaclass.MetaMethodIndex$Entry; 14: 553901 336111768 [I 15: 5044587 322853568 java.lang.invoke.BoundMethodHandle$Species\_L4IL4 16: 1781466 295495928 [Ljava.util.HashMap$Node; 17: 5041401 241987248 java.lang.invoke.BoundMethodHandle$Species\_L5 18: 2473877 217701176 java.lang.reflect.Method 19: 5819669 186229408 java.lang.invoke.BoundMethodHandle$Species\_L 20: 3668978 176110944 java.util.HashMap 21: 3242 173983752 [Ljava.util.concurrent.ConcurrentHashMap$Node; 22: 5215892 166908544 groovy.lang.MetaBeanProperty 23: 1749764 139981120 java.lang.reflect.Constructor 24: 3938142 126020544 java.lang.ref.WeakReference 25: 3130036 125201440 java.lang.invoke.BoundMethodHandle$Species\_L4 26: 101235 120651144 [S 27: 2434113 116837424 org.codehaus.groovy.util.ReferenceType$SoftRef 28: 1391360 100177920 java.lang.reflect.Field 29: 4172368 100136832 org.codehaus.groovy.util.SingleKeyHashMap 30: 658757 94861008 org.apache.lucene.codecs.blocktree.SegmentTermsEnumFrame is groovy cause? ---
**Author:** ![win1027](https://avatars.discourse-cdn.com/v4/letter/w/7bcc69/32.png) [@win1027](https://discuss.elastic.co/u/win1027)\ **Post date:** [July 28, 2016, 1:11pm UTC](https://discuss.elastic.co/t/es2-3-3-oom-someone-help/56574/6 "2016-07-28T13:11:01Z")
hot\_thread: 11.8% (58.9ms out of 500ms) cpu usage by thread 'elasticsearch[data-\*][index][T#14]' 3/10 snapshots sharing following 24 elements groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:265) groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:254) org.elasticsearch.script.groovy.GroovyScriptEngineService$3.run(GroovyScriptEngineService.java:191) org.elasticsearch.script.groovy.GroovyScriptEngineService$3.run(GroovyScriptEngineService.java:185) java.security.AccessController.doPrivileged(Native Method) org.elasticsearch.script.groovy.GroovyScriptEngineService.compile(GroovyScriptEngineService.java:185) org.elasticsearch.script.ScriptService.compileInternal(ScriptService.java:311) org.elasticsearch.script.ScriptService.compile(ScriptService.java:257) org.elasticsearch.script.ScriptService.executable(ScriptService.java:442) org.elasticsearch.action.update.UpdateHelper.executeScript(UpdateHelper.java:250) org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:197) org.elasticsearch.action.update.UpdateHelper.prepare(UpdateHelper.java:80) org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:174) org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:168) org.elasticsearch.action.update.TransportUpdateAction.shardOperation(TransportUpdateAction.java:66) org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(TransportInstanceSingleOperationAction.java:244) org.elasticsearch.action.support.single.instance.TransportInstanceSingleOperationAction$ShardTransportHandler.messageReceived(TransportInstanceSingleOperationAction.java:240) org.elasticsearch.transport.TransportRequestHandler.messageReceived(TransportRequestHandler.java:33) org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:75) org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.doRun(MessageChannelHandler.java:300) org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) java.lang.Thread.run(Thread.java:745) ---
**Author:** ![win1027](https://avatars.discourse-cdn.com/v4/letter/w/7bcc69/32.png) [@win1027](https://discuss.elastic.co/u/win1027)\ **Post date:** [August 2, 2016, 1:55pm UTC](https://discuss.elastic.co/t/es2-3-3-oom-someone-help/56574/7 "2016-08-02T13:55:56Z")
solution: > ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:30pm UTC](https://discuss.elastic.co/t/es2-3-3-oom-someone-help/56574/8 "2017-07-05T22:30:45Z")