GC Problem

Mark_Poole · March 27, 2014, 4:52pm

Hey guys.

I had a strange problem with a 1.1.0: I have a 3 node cluster and
yesterday one of the nodes started to run GC one after another, causing the
cluster to become unavailable until I restarted the one node.

gist.github.com

https://gist.github.com/reedox/9812169

gistfile1.txt


Time	Event
3/26/14 
6:13:07.217 PM	
[2014-03-26 18:13:07,217][WARN ][monitor.jvm              ] [node.company.net] [gc][old][740388][8643] duration [13.1s], collections [1]/[13.2s], total [13.1s]/[1.1h], memory [11.7gb]->[11.7gb]/[11.7gb], all_pools {[young] [260.9mb]->[265.8mb]/[266.2mb]}{[survivor] [0b]->[0b]/[33.2mb]}{[old] [11.4gb]->[11.4gb]/[11.4gb]}
host = node.company.net source = /var/log/elasticsearch/companyapp.log sourcetype = elasticsearch_companyapp_log
3/26/14 
6:12:53.922 PM	
[2014-03-26 18:12:53,922][INFO ][monitor.jvm              ] [node.company.net] [gc][old][740387][8642] duration [9.9s], collections [1]/[10s], total [9.9s]/[1.1h], memory [11.7gb]->[11.7gb]/[11.7gb], all_pools {[young] [266.2mb]->[260.9mb]/[266.2mb]}{[survivor] [3.5mb]->[0b]/[33.2mb]}{[old] [11.4gb]->[11.4gb]/[11.4gb]}
host = node.company.net source = /var/log/elasticsearch/companyapp.log sourcetype = elasticsearch_companyapp_log

This file has been truncated. show original

Any ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f5d5b636-ee3e-4cba-9c45-b419d2b0bd21%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Mark_Poole · March 27, 2014, 5:10pm

Apologies for the follow up, i'm running 1.0.1

On Thursday, March 27, 2014 9:52:48 AM UTC-7, Mark Poole wrote:

Hey guys.

I had a strange problem with a 1.1.0: I have a 3 node cluster and
yesterday one of the nodes started to run GC one after another, causing the
cluster to become unavailable until I restarted the one node.

gist:9812169 · GitHub

Any ideas?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2e9c233a-530b-4655-bde2-78adb6c397ad%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Binh_Ly_2 · March 27, 2014, 6:36pm

I'm assuming this log is on a single node. Looking at it, it seems like the
GC just couldn't free anything up so is it possible that you just
physically need more RAM than 11G on this node to handle the load on it?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/faa2dfa1-0586-40a7-b7e3-a68ea1a35ddb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.