Memory issues on ES client node


(Venkat Morampudi) #1

In the elastic search deployment we are seeing random client node crashed
due to out of memory exception. I got the memory dump from one of the crash
and analysed using Eclipse memory analyzer. I have attached leak suspect
report. Apparently 242 objects of
type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
are holding almost 8gb of memory. I have spent some time on source code but
couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #2

Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi venkatmorampudi@gmail.com
wrote:

In the elastic search deployment we are seeing random client node crashed
due to out of memory exception. I got the memory dump from one of the crash
and analysed using Eclipse memory analyzer. I have attached leak suspect
report. Apparently 242 objects of
type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
are holding almost 8gb of memory. I have spent some time on source code but
couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #3

Are you using anything to monitor your cluster, plugins like ElasticHQ or
Marvel?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 2 July 2014 05:58, joergprante@gmail.com joergprante@gmail.com wrote:

Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi <
venkatmorampudi@gmail.com> wrote:

In the elastic search deployment we are seeing random client node crashed
due to out of memory exception. I got the memory dump from one of the crash
and analysed using Eclipse memory analyzer. I have attached leak suspect
report. Apparently 242 objects of
type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
are holding almost 8gb of memory. I have spent some time on source code but
couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624bvS7Y9HoFmMG8NeQM-hWxwqPm9eb6viFANnsnsU_POFw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Venkat Morampudi) #4

Thanks for reply Jörg. I don't have any logs. I will try to enable them it
would but it would take some time though. If there anything in particular
that we need to enable, please let me know.

-VM

On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:

Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi <venkatm...@gmail.com
<javascript:>> wrote:

In the elastic search deployment we are seeing random client node crashed
due to out of memory exception. I got the memory dump from one of the crash
and analysed using Eclipse memory analyzer. I have attached leak suspect
report. Apparently 242 objects of
type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
are holding almost 8gb of memory. I have spent some time on source code but
couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Venkat Morampudi) #5

Hi Mark, is the any particular aspect of monitoring or monitoring in
general?

-VM

On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote:

Are you using anything to monitor your cluster, plugins like ElasticHQ or
Marvel?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 2 July 2014 05:58, joerg...@gmail.com <javascript:> <joerg...@gmail.com
<javascript:>> wrote:

Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi <venkatm...@gmail.com
<javascript:>> wrote:

In the elastic search deployment we are seeing random client node
crashed due to out of memory exception. I got the memory dump from one of
the crash and analysed using Eclipse memory analyzer. I have attached leak
suspect report. Apparently 242 objects of
type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction
are holding almost 8gb of memory. I have spent some time on source code but
couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Walkom) #6

The plugins I mentioned before give you insight into cluster state,
specifically around GC. They would be worth installing if you want to
troubleshoot this.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 2 July 2014 11:30, Venkat Morampudi venkatmorampudi@gmail.com wrote:

Hi Mark, is the any particular aspect of monitoring or monitoring in
general?

-VM

On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote:

Are you using anything to monitor your cluster, plugins like ElasticHQ or
Marvel?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 2 July 2014 05:58, joerg...@gmail.com joerg...@gmail.com wrote:

Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi venkatm...@gmail.com
wrote:

In the elastic search deployment we are seeing random client node
crashed due to out of memory exception. I got the memory dump from one of
the crash and analysed using Eclipse memory analyzer. I have attached leak
suspect report. Apparently 242 objects of type org.elasticsearch.action.
search.type.TransportSearchQueryThenFetchAction$AsyncAction are
holding almost 8gb of memory. I have spent some time on source code but
couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7E
SMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YnVud3oFFoMU7-cuqNro6Q1ML00AurMx3RZ8Eaeis0Pw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Venkat Morampudi) #7

Thanks Mark. I will look into installing the plug-ins that you suggested.

-VM

On Jul 1, 2014, at 8:11 PM, Mark Walkom markw@campaignmonitor.com wrote:

The plugins I mentioned before give you insight into cluster state, specifically around GC. They would be worth installing if you want to troubleshoot this.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 2 July 2014 11:30, Venkat Morampudi venkatmorampudi@gmail.com wrote:
Hi Mark, is the any particular aspect of monitoring or monitoring in general?

-VM

On Tuesday, July 1, 2014 3:40:23 PM UTC-7, Mark Walkom wrote:
Are you using anything to monitor your cluster, plugins like ElasticHQ or Marvel?

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com
web: www.campaignmonitor.com

On 2 July 2014 05:58, joerg...@gmail.com joerg...@gmail.com wrote:
Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi venkatm...@gmail.com wrote:
In the elastic search deployment we are seeing random client node crashed due to out of memory exception. I got the memory dump from one of the crash and analysed using Eclipse memory analyzer. I have attached leak suspect report. Apparently 242 objects of type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding almost 8gb of memory. I have spent some time on source code but couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEBDtfZ6mcFTkKPuW1DDGJ7ESMXgCgDpn%2BTEeuuAXswEg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/26a214d8-ce3a-4ab6-96e9-304800f1f9d9%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YnVud3oFFoMU7-cuqNro6Q1ML00AurMx3RZ8Eaeis0Pw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/29787865-DCEF-43F5-8DCA-B5DAD4F7324B%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #8

I'm not sure but it looks like a node tries to move some GB of document
hits around. This might have triggered timeouts at other places (probably
with node disconnects) and maybe the GB chunk is not yet GC collected, so
you see this in your heap analyzer tool.

It depends on the search results and search hits you generated if the
heaviness of the search result is expected or not, so it would be useful to
know more about your queries.

Jörg

On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi venkatmorampudi@gmail.com
wrote:

Thanks for reply Jörg. I don't have any logs. I will try to enable them it
would but it would take some time though. If there anything in particular
that we need to enable, please let me know.

-VM

On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:

Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi venkatm...@gmail.com
wrote:

In the elastic search deployment we are seeing random client node
crashed due to out of memory exception. I got the memory dump from one of
the crash and analysed using Eclipse memory analyzer. I have attached leak
suspect report. Apparently 242 objects of type org.elasticsearch.action.
search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding
almost 8gb of memory. I have spent some time on source code but couldn't
find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Venkat Morampudi) #9

It expected to nodes move huge volumes of data but what I was wondering why the objects are not being garbage collected. Also, there are 242 TransportSearchQueryThenFetchAction$AsyncAction; I don't think that kind of concurrency is not expected. I couldn't yet find from the code which object is holding those objects.

The timeouts that you are referring to, are these between client node and data nodes or client node and consumer? Is there any thing the consumer need to do to release objects.

Thanks for your time,
-VM

On Jul 2, 2014, at 7:08 AM, joergprante@gmail.com wrote:

I'm not sure but it looks like a node tries to move some GB of document hits around. This might have triggered timeouts at other places (probably with node disconnects) and maybe the GB chunk is not yet GC collected, so you see this in your heap analyzer tool.

It depends on the search results and search hits you generated if the heaviness of the search result is expected or not, so it would be useful to know more about your queries.

Jörg

On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi venkatmorampudi@gmail.com wrote:
Thanks for reply Jörg. I don't have any logs. I will try to enable them it would but it would take some time though. If there anything in particular that we need to enable, please let me know.

-VM

On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:
Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi venkatm...@gmail.com wrote:
In the elastic search deployment we are seeing random client node crashed due to out of memory exception. I got the memory dump from one of the crash and analysed using Eclipse memory analyzer. I have attached leak suspect report. Apparently 242 objects of type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding almost 8gb of memory. I have spent some time on source code but couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #10

I think this is not a concurrency problem but the cluster wanted to deliver
a huge portion of data (just guessing about such query responses because I
do not know anything about the queries on your system).

Client timeout of receiving data is around 30 secs IIRC. It only means that
it could be possible the client gave up before the transfer of the data was
done.

Jörg

On Mon, Jul 7, 2014 at 4:49 AM, Venkat Morampudi venkatmorampudi@gmail.com
wrote:

It expected to nodes move huge volumes of data but what I was wondering
why the objects are not being garbage collected. Also, there are 242
TransportSearchQueryThenFetchAction$AsyncAction; I don’t think that kind
of concurrency is not expected. I couldn’t yet find from the code which
object is holding those objects.

The timeouts that you are referring to, are these between client node and
data nodes or client node and consumer? Is there any thing the consumer
need to do to release objects.

Thanks for your time,
-VM

On Jul 2, 2014, at 7:08 AM, joergprante@gmail.com wrote:

I'm not sure but it looks like a node tries to move some GB of document
hits around. This might have triggered timeouts at other places (probably
with node disconnects) and maybe the GB chunk is not yet GC collected, so
you see this in your heap analyzer tool.

It depends on the search results and search hits you generated if the
heaviness of the search result is expected or not, so it would be useful to
know more about your queries.

Jörg

On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi <
venkatmorampudi@gmail.com> wrote:

Thanks for reply Jörg. I don't have any logs. I will try to enable them
it would but it would take some time though. If there anything in
particular that we need to enable, please let me know.

-VM

On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:

Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi venkatm...@gmail.com
wrote:

In the elastic search deployment we are seeing random client node
crashed due to out of memory exception. I got the memory dump from one of
the crash and analysed using Eclipse memory analyzer. I have attached leak
suspect report. Apparently 242 objects of type org.elasticsearch.action.
search.type.TransportSearchQueryThenFetchAction$AsyncAction are
holding almost 8gb of memory. I have spent some time on source code but
couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com
https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXef75OF%3D6Xk8ZQczL-F63kqdLYzg%2BhBFd61BDd2xeng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Venkat Morampudi) #11

I am having connection the dots. How the timeout lead to out of memory exceptions?

Thanks,

-VM

On Jul 7, 2014, at 9:54 AM, joergprante@gmail.com wrote:

I think this is not a concurrency problem but the cluster wanted to deliver a huge portion of data (just guessing about such query responses because I do not know anything about the queries on your system).

Client timeout of receiving data is around 30 secs IIRC. It only means that it could be possible the client gave up before the transfer of the data was done.

Jörg

On Mon, Jul 7, 2014 at 4:49 AM, Venkat Morampudi venkatmorampudi@gmail.com wrote:
It expected to nodes move huge volumes of data but what I was wondering why the objects are not being garbage collected. Also, there are 242 TransportSearchQueryThenFetchAction$AsyncAction; I don't think that kind of concurrency is not expected. I couldn't yet find from the code which object is holding those objects.

The timeouts that you are referring to, are these between client node and data nodes or client node and consumer? Is there any thing the consumer need to do to release objects.

Thanks for your time,
-VM

On Jul 2, 2014, at 7:08 AM, joergprante@gmail.com wrote:

I'm not sure but it looks like a node tries to move some GB of document hits around. This might have triggered timeouts at other places (probably with node disconnects) and maybe the GB chunk is not yet GC collected, so you see this in your heap analyzer tool.

It depends on the search results and search hits you generated if the heaviness of the search result is expected or not, so it would be useful to know more about your queries.

Jörg

On Wed, Jul 2, 2014 at 3:29 AM, Venkat Morampudi venkatmorampudi@gmail.com wrote:
Thanks for reply Jörg. I don't have any logs. I will try to enable them it would but it would take some time though. If there anything in particular that we need to enable, please let me know.

-VM

On Tuesday, July 1, 2014 12:58:21 PM UTC-7, Jörg Prante wrote:
Do you have anything in your logs, i.e. many disconnects/reconnects?

Jörg

On Tue, Jul 1, 2014 at 7:59 PM, Venkat Morampudi venkatm...@gmail.com wrote:
In the elastic search deployment we are seeing random client node crashed due to out of memory exception. I got the memory dump from one of the crash and analysed using Eclipse memory analyzer. I have attached leak suspect report. Apparently 242 objects of type org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction are holding almost 8gb of memory. I have spent some time on source code but couldn't find anything obvious.

I would really appreciate any help with this issue.

-VM

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/37881ead-70c2-40d8-89b6-a771b2a36bdd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9930fcfd-d2d4-4f62-b8a0-8f1f989069f2%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE_Xum%2BU%3D-M-X_R93qbDdOKx-QFS2PFCbxcik-uqtpBbw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/7E54C4B5-AE5A-4E64-8199-B923E8337676%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elasticsearch/EH76o1CIeQQ/unsubscribe.
To unsubscribe from this group and all its topics, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEXef75OF%3D6Xk8ZQczL-F63kqdLYzg%2BhBFd61BDd2xeng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/37C2706F-A383-4034-955D-E84704B76947%40gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #12