# ES OutOfMemory on a 30GB index **URL:** \ **Category:** Elasticsearch\ **Created:** [May 28, 2014, 9:27pm UTC](https://discuss.elastic.co/t/es-outofmemory-on-a-30gb-index/17790 "2014-05-28T21:27:25Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [May 28, 2014, 9:27pm UTC](https://discuss.elastic.co/t/es-outofmemory-on-a-30gb-index/17790/1 "2014-05-28T21:27:25Z")
Hi Everyone, We are seeing continual OOM exceptions on one of our 1.1.0 elasticsearch clusters, the index is ~30GB, quite small. I'm trying to work out the root cause via heap dump analysis, but not having a lot of luck. I don't want to include a bunch of unnecessary info, but the stacktrace we're seeing is pasted below. Has anyone seen this before? I've been using the cluster stats and node stats APIs to try and find a smoking gun, but I'm not seeing anything that looks out of the ordinary. Any ideas? 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Failed to send error message back to client for action [search/phase/query] java.lang.OutOfMemoryError: GC overhead limit exceeded 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Actual Exception org.elasticsearch.search.query.QueryPhaseExecutionException: [eventdata][2]: q uery[ConstantScore(_:_)],from[0],size[0]: Query Failed [Failed to execute main query] at org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:1 27) at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623) at org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612) at org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded -- _Important Notice:_ The information contained in or attached to this email message is confidential and proprietary information of RedOwl Analytics, Inc., and by opening this email or any attachment the recipient agrees to keep such information strictly confidential and not to use or disclose the information other than as expressly authorized by RedOwl Analytics, Inc. If you are not the intended recipient, please be aware that any use, printing, copying, disclosure, dissemination, or the taking of any act in reliance on this communication or the information contained herein is strictly prohibited. If you think that you have received this email message in error, please delete it and notify the sender. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 28, 2014, 10:22pm UTC](https://discuss.elastic.co/t/es-outofmemory-on-a-30gb-index/17790/2 "2014-05-28T22:22:29Z")
Can you provide some specs on your cluster, OS, RAM, heap, disk, java and ES versions? Are you using parent/child relationships, TTLs, large facet or other queries? (Also, your elaborate legalese signature is kind of moot given you're posting to a public mailing list :p) Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 29 May 2014 07:27, Paul Sanwald [paul@redowlanalytics.com](mailto:paul@redowlanalytics.com) wrote: > Hi Everyone, > We are seeing continual OOM exceptions on one of our 1.1.0 > elasticsearch clusters, the index is ~30GB, quite small. I'm trying to work > out the root cause via heap dump analysis, but not having a lot of luck. I > don't want to include a bunch of unnecessary info, but the stacktrace we're > seeing is pasted below. Has anyone seen this before? I've been using the > cluster stats and node stats APIs to try and find a smoking gun, but I'm > not seeing anything that looks out of the ordinary. > > Any ideas? > > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Failed to send error > message back to client for action [search/phase/query] > java.lang.OutOfMemoryError: GC overhead limit exceeded > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Actual Exception > org.elasticsearch.search.query.QueryPhaseExecutionException: > [eventdata][2]: q > uery[ConstantScore(_:_)],from[0],size[0]: Query Failed [Failed to execute > main > query] > at > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:1 > 27) > at > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) > at > org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623) > at > org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612) > at > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:722) > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > _Important Notice:_ The information contained in or attached to this > email message is confidential and proprietary information of RedOwl > Analytics, Inc., and by opening this email or any attachment the recipient > agrees to keep such information strictly confidential and not to use or > disclose the information other than as expressly authorized by RedOwl > Analytics, Inc. If you are not the intended recipient, please be aware > that any use, printing, copying, disclosure, dissemination, or the taking > of any act in reliance on this communication or the information contained > herein is strictly prohibited. If you think that you have received this > email message in error, please delete it and notify the sender. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624b9vRvomsmvN0LVbTNgg4pgtPvc7gRjB-7L0GqJtG02ug%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624b9vRvomsmvN0LVbTNgg4pgtPvc7gRjB-7L0GqJtG02ug%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [May 28, 2014, 10:33pm UTC](https://discuss.elastic.co/t/es-outofmemory-on-a-30gb-index/17790/3 "2014-05-28T22:33:46Z")
I apologize about the signature, it's automatic. I've created a gist with the cluster node stats: > We are using 1.1.0, using aggregations a lot but nothing crazy. We run our app on much much larger indices successfully. But, the problem seems to be present itself on even basic search cases. The one thing that's different about this dataset is a lot of it is in spanish. thanks for your help! On Wednesday, May 28, 2014 6:22:59 PM UTC-4, Mark Walkom wrote: > Can you provide some specs on your cluster, OS, RAM, heap, disk, java and > ES versions? > Are you using parent/child relationships, TTLs, large facet or other > queries? > > (Also, your elaborate legalese signature is kind of moot given you're > posting to a public mailing list :p) > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 29 May 2014 07:27, Paul Sanwald \<[pa...@redowlanalytics.com](mailto:pa...@redowlanalytics.com)\ > > > wrote: > > > Hi Everyone, > > We are seeing continual OOM exceptions on one of our 1.1.0 > > elasticsearch clusters, the index is ~30GB, quite small. I'm trying to work > > out the root cause via heap dump analysis, but not having a lot of luck. I > > don't want to include a bunch of unnecessary info, but the stacktrace we're > > seeing is pasted below. Has anyone seen this before? I've been using the > > cluster stats and node stats APIs to try and find a smoking gun, but I'm > > not seeing anything that looks out of the ordinary. > > > > Any ideas? > > > > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Failed to send error > > message back to client for action [search/phase/query] > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Actual Exception > > org.elasticsearch.search.query.QueryPhaseExecutionException: > > [eventdata][2]: q > > uery[ConstantScore(_:_)],from[0],size[0]: Query Failed [Failed to execute > > main > > query] > > at > > org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:1 > > 27) > > at > > org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:257) > > at > > org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:623) > > at > > org.elasticsearch.search.action.SearchServiceTransportAction$SearchQueryTransportHandler.messageReceived(SearchServiceTransportAction.java:612) > > at > > org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler.run(MessageChannelHandler.java:270) > > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > > at java.lang.Thread.run(Thread.java:722) > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > _Important Notice:_ The information contained in or attached to this > > email message is confidential and proprietary information of RedOwl > > Analytics, Inc., and by opening this email or any attachment the recipient > > agrees to keep such information strictly confidential and not to use or > > disclose the information other than as expressly authorized by RedOwl > > Analytics, Inc. If you are not the intended recipient, please be aware > > that any use, printing, copying, disclosure, dissemination, or the taking > > of any act in reliance on this communication or the information contained > > herein is strictly prohibited. If you think that you have received this > > email message in error, please delete it and notify the sender. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- _Important Notice:_ The information contained in or attached to this email message is confidential and proprietary information of RedOwl Analytics, Inc., and by opening this email or any attachment the recipient agrees to keep such information strictly confidential and not to use or disclose the information other than as expressly authorized by RedOwl Analytics, Inc. If you are not the intended recipient, please be aware that any use, printing, copying, disclosure, dissemination, or the taking of any act in reliance on this communication or the information contained herein is strictly prohibited. If you think that you have received this email message in error, please delete it and notify the sender. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 28, 2014, 10:57pm UTC](https://discuss.elastic.co/t/es-outofmemory-on-a-30gb-index/17790/4 "2014-05-28T22:57:56Z")
What java version are you running, it's not in the stats gist. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 29 May 2014 08:33, Paul Sanwald [paul@redowlanalytics.com](mailto:paul@redowlanalytics.com) wrote: > I apologize about the signature, it's automatic. I've created a gist with > the cluster node stats: > [gist:e11ba02ac591757c8d92 · GitHub](https://gist.github.com/pcsanwald/e11ba02ac591757c8d92) > > We are using 1.1.0, using aggregations a lot but nothing crazy. We run our > app on much much larger indices successfully. But, the problem seems to be > present itself on even basic search cases. The one thing that's different > about this dataset is a lot of it is in spanish. > > thanks for your help! > > On Wednesday, May 28, 2014 6:22:59 PM UTC-4, Mark Walkom wrote: > > > Can you provide some specs on your cluster, OS, RAM, heap, disk, java and > > ES versions? > > Are you using parent/child relationships, TTLs, large facet or other > > queries? > > > > (Also, your elaborate legalese signature is kind of moot given you're > > posting to a public mailing list :p) > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 29 May 2014 07:27, Paul Sanwald [pa...@redowlanalytics.com](mailto:pa...@redowlanalytics.com) wrote: > > > > > Hi Everyone, > > > We are seeing continual OOM exceptions on one of our 1.1.0 > > > elasticsearch clusters, the index is ~30GB, quite small. I'm trying to work > > > out the root cause via heap dump analysis, but not having a lot of luck. I > > > don't want to include a bunch of unnecessary info, but the stacktrace we're > > > seeing is pasted below. Has anyone seen this before? I've been using the > > > cluster stats and node stats APIs to try and find a smoking gun, but I'm > > > not seeing anything that looks out of the ordinary. > > > > > > Any ideas? > > > > > > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Failed to send error > > > message back to client for action [search/phase/query] > > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Actual Exception > > > org.elasticsearch.search.query.QueryPhaseExecutionException: > > > [eventdata][2]: q > > > uery[ConstantScore(_:_)],from[0],size[0]: Query Failed [Failed to > > > execute main > > > query] > > > at org.elasticsearch.search.query.QueryPhase.execute( > > > QueryPhase.java:1 > > > 27) > > > at org.elasticsearch.search.SearchService.executeQueryPhase( > > > SearchService.java:257) > > > at org.elasticsearch.search.action.SearchServiceTransportAction$ > > > SearchQueryTransportHandler.messageReceived( > > > SearchServiceTransportAction.java:623) > > > at org.elasticsearch.search.action.SearchServiceTransportAction$ > > > SearchQueryTransportHandler.messageReceived( > > > SearchServiceTransportAction.java:612) > > > at org.elasticsearch.transport.netty.MessageChannelHandler$ > > > RequestHandler.run(MessageChannelHandler.java:270) > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > ThreadPoolExecutor.java:1145) > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > ThreadPoolExecutor.java:615) > > > at java.lang.Thread.run(Thread.java:722) > > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > > > _Important Notice:_ The information contained in or attached to this > > > email message is confidential and proprietary information of RedOwl > > > Analytics, Inc., and by opening this email or any attachment the recipient > > > agrees to keep such information strictly confidential and not to use or > > > disclose the information other than as expressly authorized by RedOwl > > > Analytics, Inc. If you are not the intended recipient, please be aware > > > that any use, printing, copying, disclosure, dissemination, or the taking > > > of any act in reliance on this communication or the information contained > > > herein is strictly prohibited. If you think that you have received this > > > email message in error, please delete it and notify the sender. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88% > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > _Important Notice:_ The information contained in or attached to this > email message is confidential and proprietary information of RedOwl > Analytics, Inc., and by opening this email or any attachment the recipient > agrees to keep such information strictly confidential and not to use or > disclose the information other than as expressly authorized by RedOwl > Analytics, Inc. If you are not the intended recipient, please be aware > that any use, printing, copying, disclosure, dissemination, or the taking > of any act in reliance on this communication or the information contained > herein is strictly prohibited. If you think that you have received this > email message in error, please delete it and notify the sender. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624Y0KBzNUdRx0LWALTdJp9W4TOCXnt3YG0jFFBy%2BeBBR-A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624Y0KBzNUdRx0LWALTdJp9W4TOCXnt3YG0jFFBy%2BeBBR-A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [May 28, 2014, 11:11pm UTC](https://discuss.elastic.co/t/es-outofmemory-on-a-30gb-index/17790/5 "2014-05-28T23:11:05Z")
Sorry, it's Java 7: jvm: { pid: 20424 version: 1.7.0\_09-icedtea vm\_name: OpenJDK 64-Bit Server VM vm\_version: 23.7-b01 vm\_vendor: Oracle Corporation start\_time: 1401309063644 mem: { heap\_init\_in\_bytes: 1073741824 heap\_max\_in\_bytes: 10498867200 non\_heap\_init\_in\_bytes: 24313856 non\_heap\_max\_in\_bytes: 318767104 direct\_max\_in\_bytes: 10498867200 } gc\_collectors: [ PS Scavenge PS MarkSweep ] memory\_pools: [ Code Cache PS Eden Space PS Survivor Space PS Old Gen PS Perm Gen ] On Wednesday, May 28, 2014 6:58:26 PM UTC-4, Mark Walkom wrote: > What java version are you running, it's not in the stats gist. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 29 May 2014 08:33, Paul Sanwald \<[pa...@redowlanalytics.com](mailto:pa...@redowlanalytics.com)\ > > > wrote: > > > I apologize about the signature, it's automatic. I've created a gist with > > the cluster node stats: > > [gist:e11ba02ac591757c8d92 · GitHub](https://gist.github.com/pcsanwald/e11ba02ac591757c8d92) > > > > We are using 1.1.0, using aggregations a lot but nothing crazy. We run > > our app on much much larger indices successfully. But, the problem seems to > > be present itself on even basic search cases. The one thing that's > > different about this dataset is a lot of it is in spanish. > > > > thanks for your help! > > > > On Wednesday, May 28, 2014 6:22:59 PM UTC-4, Mark Walkom wrote: > > > > > Can you provide some specs on your cluster, OS, RAM, heap, disk, java > > > and ES versions? > > > Are you using parent/child relationships, TTLs, large facet or other > > > queries? > > > > > > (Also, your elaborate legalese signature is kind of moot given you're > > > posting to a public mailing list :p) > > > > > > Regards, > > > Mark Walkom > > > > > > Infrastructure Engineer > > > Campaign Monitor > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > On 29 May 2014 07:27, Paul Sanwald [pa...@redowlanalytics.com](mailto:pa...@redowlanalytics.com) wrote: > > > > > > > Hi Everyone, > > > > We are seeing continual OOM exceptions on one of our 1.1.0 > > > > elasticsearch clusters, the index is ~30GB, quite small. I'm trying to work > > > > out the root cause via heap dump analysis, but not having a lot of luck. I > > > > don't want to include a bunch of unnecessary info, but the stacktrace we're > > > > seeing is pasted below. Has anyone seen this before? I've been using the > > > > cluster stats and node stats APIs to try and find a smoking gun, but I'm > > > > not seeing anything that looks out of the ordinary. > > > > > > > > Any ideas? > > > > > > > > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Failed to send > > > > error message back to client for action [search/phase/query] > > > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Actual Exception > > > > org.elasticsearch.search.query.QueryPhaseExecutionException: > > > > [eventdata][2]: q > > > > uery[ConstantScore(_:_)],from[0],size[0]: Query Failed [Failed to > > > > execute main > > > > query] > > > > at org.elasticsearch.search.query.QueryPhase.execute( > > > > QueryPhase.java:1 > > > > 27) > > > > at org.elasticsearch.search.SearchService.executeQueryPhase( > > > > SearchService.java:257) > > > > at org.elasticsearch.search.action. > > > > SearchServiceTransportAction$SearchQueryTransportHandler. > > > > messageReceived(SearchServiceTransportAction.java:623) > > > > at org.elasticsearch.search.action. > > > > SearchServiceTransportAction$SearchQueryTransportHandler. > > > > messageReceived(SearchServiceTransportAction.java:612) > > > > at org.elasticsearch.transport.netty.MessageChannelHandler$ > > > > RequestHandler.run(MessageChannelHandler.java:270) > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > > ThreadPoolExecutor.java:1145) > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > > ThreadPoolExecutor.java:615) > > > > at java.lang.Thread.run(Thread.java:722) > > > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > > > > > _Important Notice:_ The information contained in or attached to this > > > > email message is confidential and proprietary information of RedOwl > > > > Analytics, Inc., and by opening this email or any attachment the recipient > > > > agrees to keep such information strictly confidential and not to use or > > > > disclose the information other than as expressly authorized by RedOwl > > > > Analytics, Inc. If you are not the intended recipient, please be aware > > > > that any use, printing, copying, disclosure, dissemination, or the taking > > > > of any act in reliance on this communication or the information contained > > > > herein is strictly prohibited. If you think that you have received this > > > > email message in error, please delete it and notify the sender. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88% > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > _Important Notice:_ The information contained in or attached to this > > email message is confidential and proprietary information of RedOwl > > Analytics, Inc., and by opening this email or any attachment the recipient > > agrees to keep such information strictly confidential and not to use or > > disclose the information other than as expressly authorized by RedOwl > > Analytics, Inc. If you are not the intended recipient, please be aware > > that any use, printing, copying, disclosure, dissemination, or the taking > > of any act in reliance on this communication or the information contained > > herein is strictly prohibited. If you think that you have received this > > email message in error, please delete it and notify the sender. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- _Important Notice:_ The information contained in or attached to this email message is confidential and proprietary information of RedOwl Analytics, Inc., and by opening this email or any attachment the recipient agrees to keep such information strictly confidential and not to use or disclose the information other than as expressly authorized by RedOwl Analytics, Inc. If you are not the intended recipient, please be aware that any use, printing, copying, disclosure, dissemination, or the taking of any act in reliance on this communication or the information contained herein is strictly prohibited. If you think that you have received this email message in error, please delete it and notify the sender. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c13bcdc4-7a9e-437c-b951-2d1b05f76da1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c13bcdc4-7a9e-437c-b951-2d1b05f76da1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Paul\_Sanwald\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_sanwald_2/32/881_2.png) [@Paul\_Sanwald\_2](https://discuss.elastic.co/u/Paul_Sanwald_2)\ **Post date:** [May 29, 2014, 12:49pm UTC](https://discuss.elastic.co/t/es-outofmemory-on-a-30gb-index/17790/6 "2014-05-29T12:49:09Z")
We've narrowed the problem down to a multi\_match clause in our query: {"multi\_match":{"fields":["attachments.\*.bodies"], "query":"foobar"}} This has to do with the way we've structured our index, We are searching an index that contains emails, and we are indexing attachments in the attachments.\*.bodies fields. For example, attachments.1.bodies would contain the text body of an attachment. This structure is clearly sub-optimal in terms of multi\_match queries, but I need to structure our index in some way that we can search the contents of an email and the parsed contents of its attachments, and get back the email as a result. From reading the docs, it seems like the better way to solve this is with nested types? > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. --paul On Wednesday, May 28, 2014 7:11:05 PM UTC-4, Paul Sanwald wrote: > Sorry, it's Java 7: > > jvm: { > pid: 20424 > version: 1.7.0\_09-icedtea > vm\_name: OpenJDK 64-Bit Server VM > vm\_version: 23.7-b01 > vm\_vendor: Oracle Corporation > start\_time: 1401309063644 > mem: { > heap\_init\_in\_bytes: 1073741824 > heap\_max\_in\_bytes: 10498867200 > non\_heap\_init\_in\_bytes: 24313856 > non\_heap\_max\_in\_bytes: 318767104 > direct\_max\_in\_bytes: 10498867200 > } > gc\_collectors: [ > PS Scavenge > PS MarkSweep > ] > memory\_pools: [ > Code Cache > PS Eden Space > PS Survivor Space > PS Old Gen > PS Perm Gen > ] > > On Wednesday, May 28, 2014 6:58:26 PM UTC-4, Mark Walkom wrote: > > > What java version are you running, it's not in the stats gist. > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 29 May 2014 08:33, Paul Sanwald [pa...@redowlanalytics.com](mailto:pa...@redowlanalytics.com) wrote: > > > > > I apologize about the signature, it's automatic. I've created a gist > > > with the cluster node stats: > > > [gist:e11ba02ac591757c8d92 · GitHub](https://gist.github.com/pcsanwald/e11ba02ac591757c8d92) > > > > > > We are using 1.1.0, using aggregations a lot but nothing crazy. We run > > > our app on much much larger indices successfully. But, the problem seems to > > > be present itself on even basic search cases. The one thing that's > > > different about this dataset is a lot of it is in spanish. > > > > > > thanks for your help! > > > > > > On Wednesday, May 28, 2014 6:22:59 PM UTC-4, Mark Walkom wrote: > > > > > > > Can you provide some specs on your cluster, OS, RAM, heap, disk, java > > > > and ES versions? > > > > Are you using parent/child relationships, TTLs, large facet or other > > > > queries? > > > > > > > > (Also, your elaborate legalese signature is kind of moot given you're > > > > posting to a public mailing list :p) > > > > > > > > Regards, > > > > Mark Walkom > > > > > > > > Infrastructure Engineer > > > > Campaign Monitor > > > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > > > > > On 29 May 2014 07:27, Paul Sanwald [pa...@redowlanalytics.com](mailto:pa...@redowlanalytics.com) wrote: > > > > > > > > > Hi Everyone, > > > > > We are seeing continual OOM exceptions on one of our 1.1.0 > > > > > elasticsearch clusters, the index is ~30GB, quite small. I'm trying to work > > > > > out the root cause via heap dump analysis, but not having a lot of luck. I > > > > > don't want to include a bunch of unnecessary info, but the stacktrace we're > > > > > seeing is pasted below. Has anyone seen this before? I've been using the > > > > > cluster stats and node stats APIs to try and find a smoking gun, but I'm > > > > > not seeing anything that looks out of the ordinary. > > > > > > > > > > Any ideas? > > > > > > > > > > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Failed to send > > > > > error message back to client for action [search/phase/query] > > > > > java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > > 14/05/27 20:37:08 WARN transport.netty: [Strongarm] Actual Exception > > > > > org.elasticsearch.search.query.QueryPhaseExecutionException: > > > > > [eventdata][2]: q > > > > > uery[ConstantScore(_:_)],from[0],size[0]: Query Failed [Failed to > > > > > execute main > > > > > query] > > > > > at org.elasticsearch.search.query.QueryPhase.execute( > > > > > QueryPhase.java:1 > > > > > 27) > > > > > at org.elasticsearch.search.SearchService.executeQueryPhase( > > > > > SearchService.java:257) > > > > > at org.elasticsearch.search.action. > > > > > SearchServiceTransportAction$SearchQueryTransportHandler. > > > > > messageReceived(SearchServiceTransportAction.java:623) > > > > > at org.elasticsearch.search.action. > > > > > SearchServiceTransportAction$SearchQueryTransportHandler. > > > > > messageReceived(SearchServiceTransportAction.java:612) > > > > > at org.elasticsearch.transport.netty.MessageChannelHandler$ > > > > > RequestHandler.run(MessageChannelHandler.java:270) > > > > > at java.util.concurrent.ThreadPoolExecutor.runWorker( > > > > > ThreadPoolExecutor.java:1145) > > > > > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > > > > > ThreadPoolExecutor.java:615) > > > > > at java.lang.Thread.run(Thread.java:722) > > > > > Caused by: java.lang.OutOfMemoryError: GC overhead limit exceeded > > > > > > > > > > _Important Notice:_ The information contained in or attached to this > > > > > email message is confidential and proprietary information of RedOwl > > > > > Analytics, Inc., and by opening this email or any attachment the recipient > > > > > agrees to keep such information strictly confidential and not to use or > > > > > disclose the information other than as expressly authorized by RedOwl > > > > > Analytics, Inc. If you are not the intended recipient, please be aware > > > > > that any use, printing, copying, disclosure, dissemination, or the taking > > > > > of any act in reliance on this communication or the information contained > > > > > herein is strictly prohibited. If you think that you have received this > > > > > email message in error, please delete it and notify the sender. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88% > > > > > [40googlegroups.com](http://40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e6634ad4-619f-4f24-8287-d3bc97722a88%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > _Important Notice:_ The information contained in or attached to this > > > email message is confidential and proprietary information of RedOwl > > > Analytics, Inc., and by opening this email or any attachment the recipient > > > agrees to keep such information strictly confidential and not to use or > > > disclose the information other than as expressly authorized by RedOwl > > > Analytics, Inc. If you are not the intended recipient, please be aware > > > that any use, printing, copying, disclosure, dissemination, or the taking > > > of any act in reliance on this communication or the information contained > > > herein is strictly prohibited. If you think that you have received this > > > email message in error, please delete it and notify the sender. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/13264f7f-625d-4452-9be7-82691f735963%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > _Important Notice:_ The information contained in or attached to this > email message is confidential and proprietary information of RedOwl > Analytics, Inc., and by opening this email or any attachment the recipient > agrees to keep such information strictly confidential and not to use or > disclose the information other than as expressly authorized by RedOwl > Analytics, Inc. If you are not the intended recipient, please be aware > that any use, printing, copying, disclosure, dissemination, or the taking > of any act in reliance on this communication or the information contained > herein is strictly prohibited. If you think that you have received this > email message in error, please delete it and notify the sender. -- _Important Notice:_ The information contained in or attached to this email message is confidential and proprietary information of RedOwl Analytics, Inc., and by opening this email or any attachment the recipient agrees to keep such information strictly confidential and not to use or disclose the information other than as expressly authorized by RedOwl Analytics, Inc. If you are not the intended recipient, please be aware that any use, printing, copying, disclosure, dissemination, or the taking of any act in reliance on this communication or the information contained herein is strictly prohibited. If you think that you have received this email message in error, please delete it and notify the sender. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2d383538-2813-44ac-a50c-8649b4a91bf8%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/2d383538-2813-44ac-a50c-8649b4a91bf8%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:26am UTC](https://discuss.elastic.co/t/es-outofmemory-on-a-30gb-index/17790/7 "2017-07-06T01:26:08Z")