# Memory issues on ES client node **URL:** \ **Category:** Elasticsearch\ **Created:** [July 1, 2014, 5:59pm UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420 "2014-07-01T17:59:50Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Venkat\_Morampudi](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@Venkat\_Morampudi](https://discuss.elastic.co/u/Venkat_Morampudi)\ **Post date:** [July 1, 2014, 5:59pm UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/1 "2014-07-01T17:59:50Z")
In the elastic search deployment we are seeing random client node crashed due to out of memory exception. I got the memory dump from one of the crash and analysed using Eclipse memory analyzer. I have attached leak suspect report. Apparently 242 objects of type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding almost 8gb of memory. I have spent some time on source code but couldn't find anything obvious. I would really appreciate any help with this issue. -VM -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 1, 2014, 7:58pm UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/2 "2014-07-01T19:58:15Z")
Do you have anything in your logs, i.e. many disconnects/reconnects? Jörg On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com) wrote: > In the Elasticsearch deployment we are seeing random client node crashed > due to out of memory exception. I got the memory dump from one of the crash > and analysed using Eclipse memory analyzer. I have attached leak suspect > report. Apparently 242 objects of > type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction > are holding almost 8gb of memory. I have spent some time on source code but > couldn't find anything obvious. > > I would really appreciate any help with this issue. > > -VM > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 1, 2014, 10:39pm UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/3 "2014-07-01T22:39:52Z")
Are you using anything to monitor your cluster, plugins like ElasticHQ or Marvel? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 2 July 2014 05:58, [joergprante@gmail.com](mailto:joergprante@gmail.com) [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Do you have anything in your logs, i.e. many disconnects/reconnects? > > Jörg > > On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi \< > [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com)\> wrote: > > > In the Elasticsearch deployment we are seeing random client node crashed > > due to out of memory exception. I got the memory dump from one of the crash > > and analysed using Eclipse memory analyzer. I have attached leak suspect > > report. Apparently 242 objects of > > type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction > > are holding almost 8gb of memory. I have spent some time on source code but > > couldn't find anything obvious. > > > > I would really appreciate any help with this issue. > > > > -VM > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624bvS7Y9HoFmMG8NeQM-hWxwqPm9eb6viFANnsnsU\_POFw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624bvS7Y9HoFmMG8NeQM-hWxwqPm9eb6viFANnsnsU_POFw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Venkat\_Morampudi](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@Venkat\_Morampudi](https://discuss.elastic.co/u/Venkat_Morampudi)\ **Post date:** [July 2, 2014, 1:29am UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/4 "2014-07-02T01:29:35Z")
Thanks for reply Jörg. I don't have any logs. I will try to enable them it would but it would take some time though. If there anything in particular that we need to enable, please let me know. -VM On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote: > Do you have anything in your logs, i.e. many disconnects/reconnects? > > Jörg > > On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi \<[venkatm...@gmail.com](mailto:venkatm...@gmail.com) > \\> wrote: > > > In the Elasticsearch deployment we are seeing random client node crashed > > due to out of memory exception. I got the memory dump from one of the crash > > and analysed using Eclipse memory analyzer. I have attached leak suspect > > report. Apparently 242 objects of > > type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction > > are holding almost 8gb of memory. I have spent some time on source code but > > couldn't find anything obvious. > > > > I would really appreciate any help with this issue. > > > > -VM > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Venkat\_Morampudi](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@Venkat\_Morampudi](https://discuss.elastic.co/u/Venkat_Morampudi)\ **Post date:** [July 2, 2014, 1:30am UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/5 "2014-07-02T01:30:48Z")
Hi Mark, is the any particular aspect of monitoring or monitoring in general? -VM On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote: > Are you using anything to monitor your cluster, plugins like ElasticHQ or > Marvel? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) \ > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 2 July 2014 05:58, [joerg...@gmail.com](mailto:joerg...@gmail.com) \ \<[joerg...@gmail.com](mailto:joerg...@gmail.com) > \\> wrote: > > > Do you have anything in your logs, i.e. many disconnects/reconnects? > > > > Jörg > > > > On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi \<[venkatm...@gmail.com](mailto:venkatm...@gmail.com) > > \\> wrote: > > > > > In the Elasticsearch deployment we are seeing random client node > > > crashed due to out of memory exception. I got the memory dump from one of > > > the crash and analysed using Eclipse memory analyzer. I have attached leak > > > suspect report. Apparently 242 objects of > > > type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction > > > are holding almost 8gb of memory. I have spent some time on source code but > > > couldn't find anything obvious. > > > > > > I would really appreciate any help with this issue. > > > > > > -VM > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 2, 2014, 3:11am UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/6 "2014-07-02T03:11:58Z")
The plugins I mentioned before give you insight into cluster state, specifically around GC. They would be worth installing if you want to troubleshoot this. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 2 July 2014 11:30, Venkat Morampudi [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com) wrote: > Hi Mark, is the any particular aspect of monitoring or monitoring in > general? > > -VM > > On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote: > > > Are you using anything to monitor your cluster, plugins like ElasticHQ or > > Marvel? > > > > Regards, > > Mark Walkom > > > > Infrastructure Engineer > > Campaign Monitor > > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > > > On 2 July 2014 05:58, [joerg...@gmail.com](mailto:joerg...@gmail.com) [joerg...@gmail.com](mailto:joerg...@gmail.com) wrote: > > > > > Do you have anything in your logs, i.e. many disconnects/reconnects? > > > > > > Jörg > > > > > > On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi [venkatm...@gmail.com](mailto:venkatm...@gmail.com) > > > wrote: > > > > > > > In the Elasticsearch deployment we are seeing random client node > > > > crashed due to out of memory exception. I got the memory dump from one of > > > > the crash and analysed using Eclipse memory analyzer. I have attached leak > > > > suspect report. Apparently 242 objects of type org.elasticsearch.action. > > > > search.type.TransportSearchQueryThenFetchAction$AsyncAction are > > > > holding almost 8gb of memory. I have spent some time on source code but > > > > couldn't find anything obvious. > > > > > > > > I would really appreciate any help with this issue. > > > > > > > > -VM > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7E > > > SMXgCgDpn%2BTEeuuAXswEg%[40mail.gmail.com](http://40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YnVud3oFFoMU7-cuqNro6Q1ML00AurMx3RZ8Eaeis0Pw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YnVud3oFFoMU7-cuqNro6Q1ML00AurMx3RZ8Eaeis0Pw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Venkat\_Morampudi](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@Venkat\_Morampudi](https://discuss.elastic.co/u/Venkat_Morampudi)\ **Post date:** [July 2, 2014, 3:32am UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/7 "2014-07-02T03:32:05Z")
Thanks Mark. I will look into installing the plug-ins that you suggested. -VM On Jul 1, 2014, at 8:11 PM, Mark Walkom [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) wrote: > The plugins I mentioned before give you insight into cluster state, specifically around GC. They would be worth installing if you want to troubleshoot this. > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 2 July 2014 11:30, Venkat Morampudi [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com) wrote: > Hi Mark, is the any particular aspect of monitoring or monitoring in general? > > -VM > > On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote: > Are you using anything to monitor your cluster, plugins like ElasticHQ or Marvel? > > Regards, > Mark Walkom > > Infrastructure Engineer > Campaign Monitor > email: [ma...@campaignmonitor.com](mailto:ma...@campaignmonitor.com) > web: [www.campaignmonitor.com](http://www.campaignmonitor.com) > > On 2 July 2014 05:58, [joerg...@gmail.com](mailto:joerg...@gmail.com) [joerg...@gmail.com](mailto:joerg...@gmail.com) wrote: > Do you have anything in your logs, i.e. many disconnects/reconnects? > > Jörg > > On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi [venkatm...@gmail.com](mailto:venkatm...@gmail.com) wrote: > In the Elasticsearch deployment we are seeing random client node crashed due to out of memory exception. I got the memory dump from one of the crash and analysed using Eclipse memory analyzer. I have attached leak suspect report. Apparently 242 objects of type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding almost 8gb of memory. I have spent some time on source code but couldn't find anything obvious. > > I would really appreciate any help with this issue. > > -VM > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YnVud3oFFoMU7-cuqNro6Q1ML00AurMx3RZ8Eaeis0Pw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YnVud3oFFoMU7-cuqNro6Q1ML00AurMx3RZ8Eaeis0Pw%40mail.gmail.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/29787865-DCEF-43F5-8DCA-B5DAD4F7324B%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/29787865-DCEF-43F5-8DCA-B5DAD4F7324B%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 2, 2014, 2:08pm UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/8 "2014-07-02T14:08:59Z")
I'm not sure but it looks like a node tries to move some GB of document hits around. This might have triggered timeouts at other places (probably with node disconnects) and maybe the GB chunk is not yet GC collected, so you see this in your heap analyzer tool. It depends on the search results and search hits you generated if the heaviness of the search result is expected or not, so it would be useful to know more about your queries. Jörg On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com) wrote: > Thanks for reply Jörg. I don't have any logs. I will try to enable them it > would but it would take some time though. If there anything in particular > that we need to enable, please let me know. > > -VM > > On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote: > > > Do you have anything in your logs, i.e. many disconnects/reconnects? > > > > Jörg > > > > On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi [venkatm...@gmail.com](mailto:venkatm...@gmail.com) > > wrote: > > > > > In the Elasticsearch deployment we are seeing random client node > > > crashed due to out of memory exception. I got the memory dump from one of > > > the crash and analysed using Eclipse memory analyzer. I have attached leak > > > suspect report. Apparently 242 objects of type org.elasticsearch.action. > > > search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding > > > almost 8gb of memory. I have spent some time on source code but couldn't > > > find anything obvious. > > > > > > I would really appreciate any help with this issue. > > > > > > -VM > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd% > > > [40googlegroups.com](http://40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE\_Xum%2BU%3D-M-X\_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Venkat\_Morampudi](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@Venkat\_Morampudi](https://discuss.elastic.co/u/Venkat_Morampudi)\ **Post date:** [July 7, 2014, 2:49am UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/9 "2014-07-07T02:49:23Z")
It expected to nodes move huge volumes of data but what I was wondering why the objects are not being garbage collected. Also, there are 242 TransportSearchQueryThenFetchAction$AsyncAction; I don't think that kind of concurrency is not expected. I couldn't yet find from the code which object is holding those objects. The timeouts that you are referring to, are these between client node and data nodes or client node and consumer? Is there any thing the consumer need to do to release objects. Thanks for your time, -VM On Jul 2, 2014, at 7:08 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > I'm not sure but it looks like a node tries to move some GB of document hits around. This might have triggered timeouts at other places (probably with node disconnects) and maybe the GB chunk is not yet GC collected, so you see this in your heap analyzer tool. > > It depends on the search results and search hits you generated if the heaviness of the search result is expected or not, so it would be useful to know more about your queries. > > Jörg > > On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com) wrote: > Thanks for reply Jörg. I don't have any logs. I will try to enable them it would but it would take some time though. If there anything in particular that we need to enable, please let me know. > > -VM > > On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote: > Do you have anything in your logs, i.e. many disconnects/reconnects? > > Jörg > > On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi [venkatm...@gmail.com](mailto:venkatm...@gmail.com) wrote: > In the Elasticsearch deployment we are seeing random client node crashed due to out of memory exception. I got the memory dump from one of the crash and analysed using Eclipse memory analyzer. I have attached leak suspect report. Apparently 242 objects of type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding almost 8gb of memory. I have spent some time on source code but couldn't find anything obvious. > > I would really appreciate any help with this issue. > > -VM > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE\_Xum%2BU%3D-M-X\_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 7, 2014, 4:54pm UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/10 "2014-07-07T16:54:09Z")
I think this is not a concurrency problem but the cluster wanted to deliver a huge portion of data (just guessing about such query responses because I do not know anything about the queries on your system). Client timeout of receiving data is around 30 secs IIRC. It only means that it could be possible the client gave up before the transfer of the data was done. Jörg On Mon, Jul 7, 2014 at 4:49 AM, Venkat Morampudi [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com) wrote: > It expected to nodes move huge volumes of data but what I was wondering > why the objects are not being garbage collected. Also, there are 242 > TransportSearchQueryThenFetchAction$AsyncAction; I don’t think that kind > of concurrency is not expected. I couldn’t yet find from the code which > object is holding those objects. > > The timeouts that you are referring to, are these between client node and > data nodes or client node and consumer? Is there any thing the consumer > need to do to release objects. > > Thanks for your time, > -VM > > On Jul 2, 2014, at 7:08 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > > I'm not sure but it looks like a node tries to move some GB of document > hits around. This might have triggered timeouts at other places (probably > with node disconnects) and maybe the GB chunk is not yet GC collected, so > you see this in your heap analyzer tool. > > It depends on the search results and search hits you generated if the > heaviness of the search result is expected or not, so it would be useful to > know more about your queries. > > Jörg > > On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi \< > [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com)\> wrote: > > > Thanks for reply Jörg. I don't have any logs. I will try to enable them > > it would but it would take some time though. If there anything in > > particular that we need to enable, please let me know. > > > > -VM > > > > On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote: > > > > > Do you have anything in your logs, i.e. many disconnects/reconnects? > > > > > > Jörg > > > > > > On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi [venkatm...@gmail.com](mailto:venkatm...@gmail.com) > > > wrote: > > > > > > > In the Elasticsearch deployment we are seeing random client node > > > > crashed due to out of memory exception. I got the memory dump from one of > > > > the crash and analysed using Eclipse memory analyzer. I have attached leak > > > > suspect report. Apparently 242 objects of type org.elasticsearch.action. > > > > search.type.TransportSearchQueryThenFetchAction$AsyncAction are > > > > holding almost 8gb of memory. I have spent some time on source code but > > > > couldn't find anything obvious. > > > > > > > > I would really appreciate any help with this issue. > > > > > > > > -VM > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE\_Xum%2BU%3D-M-X\_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE\_Xum%2BU%3D-M-X\_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXef75OF%3D6Xk8ZQczL-F63kqdLYzg%2BhBFd61BDd2xeng%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXef75OF%3D6Xk8ZQczL-F63kqdLYzg%2BhBFd61BDd2xeng%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Venkat\_Morampudi](https://avatars.discourse-cdn.com/v4/letter/v/3da27b/32.png) [@Venkat\_Morampudi](https://discuss.elastic.co/u/Venkat_Morampudi)\ **Post date:** [July 10, 2014, 3:43am UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/11 "2014-07-10T03:43:22Z")
I am having connection the dots. How the timeout lead to out of memory exceptions? Thanks, -VM On Jul 7, 2014, at 9:54 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > I think this is not a concurrency problem but the cluster wanted to deliver a huge portion of data (just guessing about such query responses because I do not know anything about the queries on your system). > > Client timeout of receiving data is around 30 secs IIRC. It only means that it could be possible the client gave up before the transfer of the data was done. > > Jörg > > On Mon, Jul 7, 2014 at 4:49 AM, Venkat Morampudi [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com) wrote: > It expected to nodes move huge volumes of data but what I was wondering why the objects are not being garbage collected. Also, there are 242 TransportSearchQueryThenFetchAction$AsyncAction; I don't think that kind of concurrency is not expected. I couldn't yet find from the code which object is holding those objects. > > The timeouts that you are referring to, are these between client node and data nodes or client node and consumer? Is there any thing the consumer need to do to release objects. > > Thanks for your time, > -VM > > On Jul 2, 2014, at 7:08 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > > > I'm not sure but it looks like a node tries to move some GB of document hits around. This might have triggered timeouts at other places (probably with node disconnects) and maybe the GB chunk is not yet GC collected, so you see this in your heap analyzer tool. > > > > It depends on the search results and search hits you generated if the heaviness of the search result is expected or not, so it would be useful to know more about your queries. > > > > Jörg > > > > On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi [venkatmorampudi@gmail.com](mailto:venkatmorampudi@gmail.com) wrote: > > Thanks for reply Jörg. I don't have any logs. I will try to enable them it would but it would take some time though. If there anything in particular that we need to enable, please let me know. > > > > -VM > > > > On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote: > > Do you have anything in your logs, i.e. many disconnects/reconnects? > > > > Jörg > > > > On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi [venkatm...@gmail.com](mailto:venkatm...@gmail.com) wrote: > > In the Elasticsearch deployment we are seeing random client node crashed due to out of memory exception. I got the memory dump from one of the crash and analysed using Eclipse memory analyzer. I have attached leak suspect report. Apparently 242 objects of type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding almost 8gb of memory. I have spent some time on source code but couldn't find anything obvious. > > > > I would really appreciate any help with this issue. > > > > -VM > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com). > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE\_Xum%2BU%3D-M-X\_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com). > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit [https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe](https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe). > To unsubscribe from this group and all its topics, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXef75OF%3D6Xk8ZQczL-F63kqdLYzg%2BhBFd61BDd2xeng%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXef75OF%3D6Xk8ZQczL-F63kqdLYzg%2BhBFd61BDd2xeng%40mail.gmail.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/37C2706F-A383-4034-955D-E84704B76947%40gmail.com](https://groups.google.com/d/msgid/elasticsearch/37C2706F-A383-4034-955D-E84704B76947%40gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:16am UTC](https://discuss.elastic.co/t/memory-issues-on-es-client-node/18420/12 "2017-07-06T01:16:54Z")