# ES 0.20.0 hangs regularly **URL:** \ **Category:** Elasticsearch\ **Created:** [October 31, 2013, 6:42pm UTC](https://discuss.elastic.co/t/es-0-20-0-hangs-regularly/14199 "2013-10-31T18:42:20Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![T\_Vinod\_Gupta](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@T\_Vinod\_Gupta](https://discuss.elastic.co/u/T_Vinod_Gupta)\ **Post date:** [October 31, 2013, 6:42pm UTC](https://discuss.elastic.co/t/es-0-20-0-hangs-regularly/14199/1 "2013-10-31T18:42:20Z")
hi, i have running cluster of 3 nodes (ES 0.20.0) and once a day or 2, it hangs for me.. by hang, i mean curl command for health check or get or search hangs forever. the log files don't have any clue on this. what can i do to debug this further? will upgrading to lucene 4 based versions help? thanks -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [November 1, 2013, 6:27am UTC](https://discuss.elastic.co/t/es-0-20-0-hangs-regularly/14199/2 "2013-11-01T06:27:20Z")
Is there truly nothing in the logs? How about frequent garbage collections? Do you have any monitoring? Without knowing anything more, I would strongly suggest upgrading to the latest release. The memory improvements in 0.90.x are truly remarkable thanks to Lucene 4 and other changes. I am not one to upgrade frequently (I am still on version 0.90.2), but the Lucene 4 based version is the way to go. Cheers, Ivan On Thu, Oct 31, 2013 at 11:42 AM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > hi, > i have running cluster of 3 nodes (ES 0.20.0) and once a day or 2, it > hangs for me.. by hang, i mean curl command for health check or get or > search hangs forever. the log files don't have any clue on this. > > what can i do to debug this further? will upgrading to lucene 4 based > versions help? > > thanks > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![T\_Vinod\_Gupta](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@T\_Vinod\_Gupta](https://discuss.elastic.co/u/T_Vinod_Gupta)\ **Post date:** [November 1, 2013, 6:56am UTC](https://discuss.elastic.co/t/es-0-20-0-hangs-regularly/14199/3 "2013-11-01T06:56:30Z")
thanks for the response.. i should have provided more details - by nothing in the logs, i mean nothing around the time of hang. i have a monit task to run the curl based health check every minute, so i know when it hangs. there are other indicators in the logs - i see long gc times, exceptions.. and occasionally, it would not just hang, but crash fully due to OOM. here are some of the messages - _[2013-11-01 00:00:12,910][WARN][monitor.jvm] [Marrina] [gc][ParNew][54839][23098] duration [3.7s], collections [2]/[20.7s], total [3.7s]/[40.7m], memory [7.9gb]-\>[7.8gb]/[7.9gb], all\_pools {[Code Cache] [12.4mb]-\>[12.4mb]/[48mb]}{[Par Eden Space] [3.3mb]-\>[13.1mb]/[66.5mb]}{[Par Survivor Space] [7mb]-\>[0b]/[8.3mb]}{[CMS Old Gen] [7.8gb]-\>[7.8gb]/[7.9gb]}{[CMS Perm Gen] [44.7mb]-\>[44.7mb]/[168mb]}_ \* \* _[2013-11-01 03:10:33,244][WARN][search.action] [Marrina] Failed tosend release search context_ _org.elasticsearch.transport.NodeDisconnectedException: [Star-Lord][inet[/10.6.14.94:9300]][search/freeContext] disconnected_ _[2013-11-01 03:10:35,293][WARN][search.action] [Marrina] Failed to send release search context_ _org.elasticsearch.transport.SendRequestTransportException: [Star-Lord][inet[/10.6.14.94:9300]][search/freeContext]_ is it because i have more data and operations than what the nodes i have can support. its a 3 node cluster (only 2 data nodes though) running on aws ec2 m1.xlarge with 8GB dedicated heap on each node. thanks On Thu, Oct 31, 2013 at 11:27 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > Is there truly nothing in the logs? How about frequent garbage > collections? Do you have any monitoring? > > Without knowing anything more, I would strongly suggest upgrading to the > latest release. The memory improvements in 0.90.x are truly remarkable > thanks to Lucene 4 and other changes. I am not one to upgrade frequently (I > am still on version 0.90.2), but the Lucene 4 based version is the way to > go. > > Cheers, > > Ivan > > On Thu, Oct 31, 2013 at 11:42 AM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > > > hi, > > i have running cluster of 3 nodes (ES 0.20.0) and once a day or 2, it > > hangs for me.. by hang, i mean curl command for health check or get or > > search hangs forever. the log files don't have any clue on this. > > > > what can i do to debug this further? will upgrading to lucene 4 based > > versions help? > > > > thanks > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![T\_Vinod\_Gupta](https://avatars.discourse-cdn.com/v4/letter/t/fbc32d/32.png) [@T\_Vinod\_Gupta](https://discuss.elastic.co/u/T_Vinod_Gupta)\ **Post date:** [November 1, 2013, 6:59am UTC](https://discuss.elastic.co/t/es-0-20-0-hangs-regularly/14199/4 "2013-11-01T06:59:44Z")
also, i see lot of messages such as these in the log file - [2013-11-01 00:04:06,534][WARN][transport] [Star-Lord] Received response for a request that has timed out, sent [59098ms] ago, timed out [29098ms] ago, action [discovery/zen/fd/masterPing], node [[Marrina][NuHNT0fXTWa33\_t6qk2tGg][inet[/10.241.58.143:9300]]], id [15642] thanks On Thu, Oct 31, 2013 at 11:56 PM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > thanks for the response.. i should have provided more details - by nothing > in the logs, i mean nothing around the time of hang. i have a monit task to > run the curl based health check every minute, so i know when it hangs. > > there are other indicators in the logs - i see long gc times, exceptions.. > and occasionally, it would not just hang, but crash fully due to OOM. here > are some of the messages - > > _[2013-11-01 00:00:12,910][WARN][monitor.jvm] [Marrina] > [gc][ParNew][54839][23098] duration [3.7s], collections [2]/[20.7s], total > [3.7s]/[40.7m], memory [7.9gb]-\>[7.8gb]/[7.9gb], all\_pools {[Code Cache] > [12.4mb]-\>[12.4mb]/[48mb]}{[Par Eden Space] > [3.3mb]-\>[13.1mb]/[66.5mb]}{[Par Survivor Space] [7mb]-\>[0b]/[8.3mb]}{[CMS > Old Gen] [7.8gb]-\>[7.8gb]/[7.9gb]}{[CMS Perm Gen] > [44.7mb]-\>[44.7mb]/[168mb]}_ > \* > \* > _[2013-11-01 03:10:33,244][WARN][search.action] [Marrina] > Failed tosend release search context_ > _org.elasticsearch.transport.NodeDisconnectedException: > [Star-Lord][inet[/10.6.14.94:9300]][search/freeContext] disconnected_ > _[2013-11-01 03:10:35,293][WARN][search.action] [Marrina] > Failed to send release search context_ > _org.elasticsearch.transport.SendRequestTransportException: > [Star-Lord][inet[/10.6.14.94:9300]][search/freeContext]_ > > is it because i have more data and operations than what the nodes i have > can support. its a 3 node cluster (only 2 data nodes though) running on aws > ec2 m1.xlarge with 8GB dedicated heap on each node. > > thanks > > On Thu, Oct 31, 2013 at 11:27 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > Is there truly nothing in the logs? How about frequent garbage > > collections? Do you have any monitoring? > > > > Without knowing anything more, I would strongly suggest upgrading to the > > latest release. The memory improvements in 0.90.x are truly remarkable > > thanks to Lucene 4 and other changes. I am not one to upgrade frequently (I > > am still on version 0.90.2), but the Lucene 4 based version is the way to > > go. > > > > Cheers, > > > > Ivan > > > > On Thu, Oct 31, 2013 at 11:42 AM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > > > > > hi, > > > i have running cluster of 3 nodes (ES 0.20.0) and once a day or 2, it > > > hangs for me.. by hang, i mean curl command for health check or get or > > > search hangs forever. the log files don't have any clue on this. > > > > > > what can i do to debug this further? will upgrading to lucene 4 based > > > versions help? > > > > > > thanks > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [November 1, 2013, 5:04pm UTC](https://discuss.elastic.co/t/es-0-20-0-hangs-regularly/14199/5 "2013-11-01T17:04:11Z")
The servers are too busy doing garbage collection and cannot respond to ping requests. There are a few ways to resolve this issue, optimizing your queries/tuning JVM and cache settings, but upgrading elasticsearch is the most sensible IMHO. Elasticsearch 1.0 will not be out until 2014, but I rather upgrade twice and not have to endure cluster hiccups due to memory pressure. Ivan On Thu, Oct 31, 2013 at 11:59 PM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > also, i see lot of messages such as these in the log file - > [2013-11-01 00:04:06,534][WARN][transport] [Star-Lord] > Received response for a request that has timed out, sent [59098ms] ago, > timed out [29098ms] ago, action [discovery/zen/fd/masterPing], node > [[Marrina][NuHNT0fXTWa33\_t6qk2tGg][inet[/10.241.58.143:9300]]], id [15642] > > thanks > > On Thu, Oct 31, 2013 at 11:56 PM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > > > thanks for the response.. i should have provided more details - by > > nothing in the logs, i mean nothing around the time of hang. i have a monit > > task to run the curl based health check every minute, so i know when it > > hangs. > > > > there are other indicators in the logs - i see long gc times, > > exceptions.. and occasionally, it would not just hang, but crash fully due > > to OOM. here are some of the messages - > > > > _[2013-11-01 00:00:12,910][WARN][monitor.jvm] [Marrina] > > [gc][ParNew][54839][23098] duration [3.7s], collections [2]/[20.7s], total > > [3.7s]/[40.7m], memory [7.9gb]-\>[7.8gb]/[7.9gb], all\_pools {[Code Cache] > > [12.4mb]-\>[12.4mb]/[48mb]}{[Par Eden Space] > > [3.3mb]-\>[13.1mb]/[66.5mb]}{[Par Survivor Space] [7mb]-\>[0b]/[8.3mb]}{[CMS > > Old Gen] [7.8gb]-\>[7.8gb]/[7.9gb]}{[CMS Perm Gen] > > [44.7mb]-\>[44.7mb]/[168mb]}_ > > \* > > \* > > _[2013-11-01 03:10:33,244][WARN][search.action] [Marrina] > > Failed tosend release search context_ > > _org.elasticsearch.transport.NodeDisconnectedException: > > [Star-Lord][inet[/10.6.14.94:9300]][search/freeContext] disconnected_ > > _[2013-11-01 03:10:35,293][WARN][search.action] [Marrina] > > Failed to send release search context_ > > _org.elasticsearch.transport.SendRequestTransportException: > > [Star-Lord][inet[/10.6.14.94:9300]][search/freeContext]_ > > > > is it because i have more data and operations than what the nodes i have > > can support. its a 3 node cluster (only 2 data nodes though) running on aws > > ec2 m1.xlarge with 8GB dedicated heap on each node. > > > > thanks > > > > On Thu, Oct 31, 2013 at 11:27 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote: > > > > > Is there truly nothing in the logs? How about frequent garbage > > > collections? Do you have any monitoring? > > > > > > Without knowing anything more, I would strongly suggest upgrading to the > > > latest release. The memory improvements in 0.90.x are truly remarkable > > > thanks to Lucene 4 and other changes. I am not one to upgrade frequently (I > > > am still on version 0.90.2), but the Lucene 4 based version is the way to > > > go. > > > > > > Cheers, > > > > > > Ivan > > > > > > On Thu, Oct 31, 2013 at 11:42 AM, T Vinod Gupta [tvinod@readypulse.com](mailto:tvinod@readypulse.com)wrote: > > > > > > > hi, > > > > i have running cluster of 3 nodes (ES 0.20.0) and once a day or 2, it > > > > hangs for me.. by hang, i mean curl command for health check or get or > > > > search hangs forever. the log files don't have any clue on this. > > > > > > > > what can i do to debug this further? will upgrading to lucene 4 based > > > > versions help? > > > > > > > > thanks > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [November 1, 2013, 7:11pm UTC](https://discuss.elastic.co/t/es-0-20-0-hangs-regularly/14199/6 "2013-11-01T19:11:15Z")
Without knowing anything about your queries and your memory considerations, it is not easy to give advice. From what it seems, you may also have to check JVM settings to be sure. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:09am UTC](https://discuss.elastic.co/t/es-0-20-0-hangs-regularly/14199/7 "2017-07-06T02:09:15Z")