# 1.0.0.Beta2 OOM logs **URL:** \ **Category:** Elasticsearch\ **Created:** [December 18, 2013, 3:55pm UTC](https://discuss.elastic.co/t/1-0-0-beta2-oom-logs/14936 "2013-12-18T15:55:50Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 18, 2013, 3:55pm UTC](https://discuss.elastic.co/t/1-0-0-beta2-oom-logs/14936/1 "2013-12-18T15:55:50Z")
I ran some heap scaling tests how to find out the required heap for my workload. My config was ES 1.0.0.Beta2 cluster, 3 RHEL 6.3 nodes, Java 1.8.0-ea JVM 25.0-b56, 4GB heap, G1 GC (and some tuning for segment merge and bulk) Workload: mixed, scan/scroll query over 1.6m docs plus term queries over 20m docs (unknown queries per second, but higher than 5000) with bulk indexing (5000 docs per second) The 4GB exercise result was OOM on all nodes after an hour run, with all kinds of error messages. The cluster restarted ok afterwards so it did not matter at all. Increasing heap to 6GB and redoing the exercise succeeded after 52 minutes. I just want to share the OOM logs with anyone who might be interested to have a look, because they are so pretty 😉 > FYI I'm considering a memory watchdog on shard level that might detect low free heap condition in time and can return warnings to the bulk client, so the bulk client might throttle, suspend, or exit the indexing cleanly, before OOMs start to break out in the cluster with all the risk of crashing shards or node dropouts. Surely not an exact science but with some heuristics it should work (e.g. below a threshold of 10mb free heap there should no execution of the indexing engine allowed) Would love to have more time for testing the exciting new 1.0.0.Beta2 features, but right now I'm just happy to run my data reconciliations successfully. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEV9HpzbT5HzdG5s0KspvQC%2Ba153-iTkue4QvRrCiTVxw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEV9HpzbT5HzdG5s0KspvQC%2Ba153-iTkue4QvRrCiTVxw%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [December 19, 2013, 7:29am UTC](https://discuss.elastic.co/t/1-0-0-beta2-oom-logs/14936/2 "2013-12-19T07:29:20Z")
Hi Jörg, Java 1.8.0-ea JVM 25.0-b56 is that java 8 or typo? /Jason On Wed, Dec 18, 2013 at 11:55 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > I ran some heap scaling tests how to find out the required heap for my > workload. > > My config was ES 1.0.0.Beta2 cluster, 3 RHEL 6.3 nodes, Java 1.8.0-ea JVM > 25.0-b56, 4GB heap, G1 GC (and some tuning for segment merge and bulk) > > Workload: mixed, scan/scroll query over 1.6m docs plus term queries over > 20m docs (unknown queries per second, but higher than 5000) with bulk > indexing (5000 docs per second) > > The 4GB exercise result was OOM on all nodes after an hour run, with all > kinds of error messages. The cluster restarted ok afterwards so it did not > matter at all. Increasing heap to 6GB and redoing the exercise succeeded > after 52 minutes. > > I just want to share the OOM logs with anyone who might be interested to > have a look, because they are so pretty 😉 > > [OOM of ES 1.0.0.Beta2 cluster 3 RHEL 6.3 nodes (zephyros, boreas, notos) Java 1.8.0-ea, JVM 25.0-b56, 4GB heap, G1 GC Workload: mixed term queries with bulk indexing (5000 docs per second) · GitHub](https://gist.github.com/jprante/8024139) > > FYI I'm considering a memory watchdog on shard level that might detect low > free heap condition in time and can return warnings to the bulk client, so > the bulk client might throttle, suspend, or exit the indexing cleanly, > before OOMs start to break out in the cluster with all the risk of crashing > shards or node dropouts. Surely not an exact science but with some > heuristics it should work (e.g. below a threshold of 10mb free heap there > should no execution of the indexing engine allowed) > > Would love to have more time for testing the exciting new 1.0.0.Beta2 > features, but right now I'm just happy to run my data reconciliations > successfully. > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEV9HpzbT5HzdG5s0KspvQC%2Ba153-iTkue4QvRrCiTVxw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEV9HpzbT5HzdG5s0KspvQC%2Ba153-iTkue4QvRrCiTVxw%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itz9DpJiegQvQoDQS2HnNfd7rqi9gc6u4-P0HvezTznVZg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itz9DpJiegQvQoDQS2HnNfd7rqi9gc6u4-P0HvezTznVZg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 19, 2013, 9:16am UTC](https://discuss.elastic.co/t/1-0-0-beta2-oom-logs/14936/3 "2013-12-19T09:16:17Z")
I'm testing Java 8, yes, for some months now. [joerg@zephyros ~]$ ls -l jdk-8\* -rw-r--r--. 1 joerg joerg 274442240 8. Dez 22:00 jdk-8-ea-bin-b114-linux-x64-31\_oct\_2013.tar [joerg@zephyros ~]$ /usr/java/jdk1.8.0/bin/java -version java version "1.8.0-ea" Java(TM) SE Runtime Environment (build 1.8.0-ea-b114) Java HotSpot(TM) 64-Bit Server VM (build 25.0-b56, mixed mode) /usr/java/jdk1.8.0/bin/java -Xms6g -Xmx6g -XX:+UseG1GC -XX:MaxGCPauseMillis=1000 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.port=3333 -Djava.rmi.server.hostname=zephyros -Delasticsearch -Des.path.home=/home/es/elasticsearch-1.0.0.Beta2 -cp :/home/es/elasticsearch-1.0.0.Beta2/lib/elasticsearch-1.0.0.Beta2.jar:/home/es/elasticsearch-1.0.0.Beta2/lib/_:/home/es/elasticsearch-1.0.0.Beta2/lib/sigar/_ org.elasticsearch.bootstrap.ElasticSearch Beware of using MVEL, it is totally broken running on a Java 8 JVM. But I would either prefer Nashorn. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDy6HDKL6S-uEt9i7\_E7nPmuFftgMAP13miBQD%3DnB3sg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDy6HDKL6S-uEt9i7_E7nPmuFftgMAP13miBQD%3DnB3sg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Jason\_Wee](https://avatars.discourse-cdn.com/v4/letter/j/7ea924/32.png) [@Jason\_Wee](https://discuss.elastic.co/u/Jason_Wee)\ **Post date:** [December 19, 2013, 10:09am UTC](https://discuss.elastic.co/t/1-0-0-beta2-oom-logs/14936/4 "2013-12-19T10:09:30Z")
Thank you for sharing, Jörg /Jason On Thu, Dec 19, 2013 at 5:16 PM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > I'm testing Java 8, yes, for some months now. > > [joerg@zephyros ~]$ ls -l jdk-8\* > -rw-r--r--. 1 joerg joerg 274442240 8. Dez 22:00 > jdk-8-ea-bin-b114-linux-x64-31\_oct\_2013.tar > > [joerg@zephyros ~]$ /usr/java/jdk1.8.0/bin/java -version > java version "1.8.0-ea" > Java(TM) SE Runtime Environment (build 1.8.0-ea-b114) > Java HotSpot(TM) 64-Bit Server VM (build 25.0-b56, mixed mode) > > /usr/java/jdk1.8.0/bin/java -Xms6g -Xmx6g -XX:+UseG1GC > -XX:MaxGCPauseMillis=1000 -Dcom.sun.management.jmxremote.ssl=false > -Dcom.sun.management.jmxremote.authenticate=false > -Dcom.sun.management.jmxremote.port=3333 > -Djava.rmi.server.hostname=zephyros -Delasticsearch > -Des.path.home=/home/es/elasticsearch-1.0.0.Beta2 -cp > :/home/es/elasticsearch-1.0.0.Beta2/lib/elasticsearch-1.0.0.Beta2.jar:/home/es/elasticsearch-1.0.0.Beta2/lib/_:/home/es/elasticsearch-1.0.0.Beta2/lib/sigar/_ > org.elasticsearch.bootstrap.Elasticsearch > > Beware of using MVEL, it is totally broken running on a Java 8 JVM. But I > would either prefer Nashorn. > > Jörg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDy6HDKL6S-uEt9i7\_E7nPmuFftgMAP13miBQD%3DnB3sg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHDy6HDKL6S-uEt9i7_E7nPmuFftgMAP13miBQD%3DnB3sg%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHO4itwR6yRzSce8iOX\_WKP2xjeuBLSLVN%2BxqjPwL%2B4cgouUKg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHO4itwR6yRzSce8iOX_WKP2xjeuBLSLVN%2BxqjPwL%2B4cgouUKg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:00am UTC](https://discuss.elastic.co/t/1-0-0-beta2-oom-logs/14936/5 "2017-07-06T02:00:17Z")
Closed on July 6, 2017, 2:00am UTC