How to enable Garbage Collector logging in an elastic search nodes

Some of the nodes are leaving the cluster from time to time..

I un commented the GC Logging options in elasticsearch.yml file as shown
and restarted my node.

################################## GC Logging
################################

monitor.jvm.gc.young.warn: 1000ms
monitor.jvm.gc.young.info: 700ms
monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s
monitor.jvm.gc.old.info: 5s
monitor.jvm.gc.old.debug: 2s

yet I'm not able to see any of the GC logging..

I came across this blog post
http://tech.taskrabbit.com/blog/2014/07/18/elasticsearch-in-production/

`ElasticSearch will note in its normal logs if any GC is happening which
effects a slow query, but we wanted more detail. Another logging option
ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM variables:

from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi

`
Where do I set the ES_USE_GC_LOGGING option in my node..
so that I can log GC events..

currently there is no gc.log file in my logs folder..
just normal cluster logs and slow index/search logs..

Thanks..

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You can set JAVA_OPTS in /etc/default/elasticsearch or
/etc/sysconfig/elasticsearch u9nder ES_JAVA_OPTS

On 3 March 2015 at 23:11, Anil Karaka anilkaraka@gmail.com wrote:

Some of the nodes are leaving the cluster from time to time..

I un commented the GC Logging options in elasticsearch.yml file as shown
and restarted my node.

################################## GC Logging
################################

monitor.jvm.gc.young.warn: 1000ms
monitor.jvm.gc.young.info: 700ms
monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s
monitor.jvm.gc.old.info: 5s
monitor.jvm.gc.old.debug: 2s

yet I'm not able to see any of the GC logging..

I came across this blog post
http://tech.taskrabbit.com/blog/2014/07/18/elasticsearch-in-production/

`ElasticSearch will note in its normal logs if any GC is happening which
effects a slow query, but we wanted more detail. Another logging option
ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM variables:

from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi

`
Where do I set the ES_USE_GC_LOGGING option in my node..
so that I can log GC events..

currently there is no gc.log file in my logs folder..
just normal cluster logs and slow index/search logs..

Thanks..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-Ot8p9iprmX73_jtP0NsZGvsYR_4wweBbD-r9i8TDewg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Can you please tell me what exactly do I need to add in
/etc/default/elasticsearch?

What should I set ES_USE_GC_LOGGING to?

I added below line in my dev ES node, and I didn't see the gc.log file in
my log folder.. I want to be sure what to do before changing anything in my
production cluster..

ES_USE_GC_LOGGING=true

And what are the GC logging options in my elasticsearch.yml do? where do
those logs appear?

Thank you.

On Wednesday, March 4, 2015 at 3:11:26 AM UTC+5:30, Mark Walkom wrote:

You can set JAVA_OPTS in /etc/default/elasticsearch or
/etc/sysconfig/elasticsearch u9nder ES_JAVA_OPTS

On 3 March 2015 at 23:11, Anil Karaka <anilk...@gmail.com <javascript:>>
wrote:

Some of the nodes are leaving the cluster from time to time..

I un commented the GC Logging options in elasticsearch.yml file as shown
and restarted my node.

################################## GC Logging
################################

monitor.jvm.gc.young.warn: 1000ms
monitor.jvm.gc.young.info: 700ms
monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s
monitor.jvm.gc.old.info: 5s
monitor.jvm.gc.old.debug: 2s

yet I'm not able to see any of the GC logging..

I came across this blog post
http://tech.taskrabbit.com/blog/2014/07/18/elasticsearch-in-production/

`ElasticSearch will note in its normal logs if any GC is happening which
effects a slow query, but we wanted more detail. Another logging option
ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM variables:

from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi

`
Where do I set the ES_USE_GC_LOGGING option in my node..
so that I can log GC events..

currently there is no gc.log file in my logs folder..
just normal cluster logs and slow index/search logs..

Thanks..

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6410585d-a272-4ff3-a58e-bacee4d53db0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Anil, you can try this:

/etc/sysconfig/elasticsearch

export ES_USE_GC_LOGGING=1
ES_JAVA_OPTS="-verbose:gc -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xloggc:"

Gavin

On Wednesday, March 4, 2015 at 9:22:50 AM UTC-5, Anil Karaka wrote:

Can you please tell me what exactly do I need to add in
/etc/default/elasticsearch?

What should I set ES_USE_GC_LOGGING to?

I added below line in my dev ES node, and I didn't see the gc.log file in
my log folder.. I want to be sure what to do before changing anything in my
production cluster..

ES_USE_GC_LOGGING=true

And what are the GC logging options in my elasticsearch.yml do? where do
those logs appear?

Thank you.

On Wednesday, March 4, 2015 at 3:11:26 AM UTC+5:30, Mark Walkom wrote:

You can set JAVA_OPTS in /etc/default/elasticsearch or
/etc/sysconfig/elasticsearch u9nder ES_JAVA_OPTS

On 3 March 2015 at 23:11, Anil Karaka anilk...@gmail.com wrote:

Some of the nodes are leaving the cluster from time to time..

I un commented the GC Logging options in elasticsearch.yml file as shown
and restarted my node.

################################## GC Logging
################################

monitor.jvm.gc.young.warn: 1000ms
monitor.jvm.gc.young.info: 700ms
monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s
monitor.jvm.gc.old.info: 5s
monitor.jvm.gc.old.debug: 2s

yet I'm not able to see any of the GC logging..

I came across this blog post
http://tech.taskrabbit.com/blog/2014/07/18/elasticsearch-in-production/

`ElasticSearch will note in its normal logs if any GC is happening
which effects a slow query, but we wanted more detail. Another logging
option ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM
variables:

from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi

`
Where do I set the ES_USE_GC_LOGGING option in my node..
so that I can log GC events..

currently there is no gc.log file in my logs folder..
just normal cluster logs and slow index/search logs..

Thanks..

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6c00bb3e-157b-4476-85b9-ba1a058d2d87%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Thanks,

I just kept those GC logging options outside the if block.. So that I don't
have to set that variable, and enabled GC logging by default.

this is in the file /usr/share/elasticsearch/bin/elasticsearch.in.sh

GC logging options

if [ "x$ES_USE_GC_LOGGING" != "x" ]; then
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"
fi

Enable gc logging by default

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"

On Saturday, March 14, 2015 at 7:18:29 AM UTC+5:30, Gavin Seng wrote:

Anil, you can try this:

/etc/sysconfig/elasticsearch

export ES_USE_GC_LOGGING=1
ES_JAVA_OPTS="-verbose:gc -XX:+PrintGCDateStamps -XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M -Xloggc:"

Gavin

On Wednesday, March 4, 2015 at 9:22:50 AM UTC-5, Anil Karaka wrote:

Can you please tell me what exactly do I need to add in
/etc/default/elasticsearch?

What should I set ES_USE_GC_LOGGING to?

I added below line in my dev ES node, and I didn't see the gc.log file in
my log folder.. I want to be sure what to do before changing anything in my
production cluster..

ES_USE_GC_LOGGING=true

And what are the GC logging options in my elasticsearch.yml do? where do
those logs appear?

Thank you.

On Wednesday, March 4, 2015 at 3:11:26 AM UTC+5:30, Mark Walkom wrote:

You can set JAVA_OPTS in /etc/default/elasticsearch or
/etc/sysconfig/elasticsearch u9nder ES_JAVA_OPTS

On 3 March 2015 at 23:11, Anil Karaka anilk...@gmail.com wrote:

Some of the nodes are leaving the cluster from time to time..

I un commented the GC Logging options in elasticsearch.yml file as
shown and restarted my node.

################################## GC Logging
################################

monitor.jvm.gc.young.warn: 1000ms
monitor.jvm.gc.young.info: 700ms
monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s
monitor.jvm.gc.old.info: 5s
monitor.jvm.gc.old.debug: 2s

yet I'm not able to see any of the GC logging..

I came across this blog post
http://tech.taskrabbit.com/blog/2014/07/18/elasticsearch-in-production/

`ElasticSearch will note in its normal logs if any GC is happening
which effects a slow query, but we wanted more detail. Another logging
option ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM
variables:

from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi

`
Where do I set the ES_USE_GC_LOGGING option in my node..
so that I can log GC events..

currently there is no gc.log file in my logs folder..
just normal cluster logs and slow index/search logs..

Thanks..

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f807b943-6aad-4923-88db-d02cdb72595f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

It's not recommended to alter the .in.sh, this file can change between
releases and your alterations can either be lost or can cause other
problems.

I'd suggest you do what Gavin mentioned.

On 1 April 2015 at 18:08, Anil Karaka anilkaraka@gmail.com wrote:

Thanks,

I just kept those GC logging options outside the if block.. So that I
don't have to set that variable, and enabled GC logging by default.

this is in the file /usr/share/elasticsearch/bin/elasticsearch.in.sh

GC logging options

if [ "x$ES_USE_GC_LOGGING" != "x" ]; then
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"
fi

Enable gc logging by default

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"

On Saturday, March 14, 2015 at 7:18:29 AM UTC+5:30, Gavin Seng wrote:

Anil, you can try this:

/etc/sysconfig/elasticsearch

export ES_USE_GC_LOGGING=1
ES_JAVA_OPTS="-verbose:gc -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
-Xloggc:"

Gavin

On Wednesday, March 4, 2015 at 9:22:50 AM UTC-5, Anil Karaka wrote:

Can you please tell me what exactly do I need to add in
/etc/default/elasticsearch?

What should I set ES_USE_GC_LOGGING to?

I added below line in my dev ES node, and I didn't see the gc.log file
in my log folder.. I want to be sure what to do before changing anything in
my production cluster..

ES_USE_GC_LOGGING=true

And what are the GC logging options in my elasticsearch.yml do? where do
those logs appear?

Thank you.

On Wednesday, March 4, 2015 at 3:11:26 AM UTC+5:30, Mark Walkom wrote:

You can set JAVA_OPTS in /etc/default/elasticsearch or
/etc/sysconfig/elasticsearch u9nder ES_JAVA_OPTS

On 3 March 2015 at 23:11, Anil Karaka anilk...@gmail.com wrote:

Some of the nodes are leaving the cluster from time to time..

I un commented the GC Logging options in elasticsearch.yml file as
shown and restarted my node.

################################## GC Logging
################################

monitor.jvm.gc.young.warn: 1000ms
monitor.jvm.gc.young.info: 700ms
monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s
monitor.jvm.gc.old.info: 5s
monitor.jvm.gc.old.debug: 2s

yet I'm not able to see any of the GC logging..

I came across this blog post http://tech.taskrabbit.com/
blog/2014/07/18/elasticsearch-in-production/

`ElasticSearch will note in its normal logs if any GC is happening
which effects a slow query, but we wanted more detail. Another logging
option ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM
variables:

from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi

`
Where do I set the ES_USE_GC_LOGGING option in my node..
so that I can log GC events..

currently there is no gc.log file in my logs folder..
just normal cluster logs and slow index/search logs..

Thanks..

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f807b943-6aad-4923-88db-d02cdb72595f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f807b943-6aad-4923-88db-d02cdb72595f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9GS%3DVhkmGncprhw4LGDko48hAGY3F15-vk-B50nayhgA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Ok, we also had to edit that file to change the default garbage collector
options and are now using G1GC. So I thought I will just change the if
condition to enable gc logs..

Any way, this is not the main problem, I'm trying to do all these things to
debug the main issue of my nodes leaving the cluster randomly.
https://groups.google.com/forum/#!topic/elasticsearch/E-aGhovVTPI here are
more details. Any help would be appreciated.

Yesterday night I increased the zen.ping.timeout from 3s to 10s of all my
machines and restarted them one by one, and yet to see if its any
improvement.
Will updating elasticsearch in my nodes fix my issues? Currently my nodes
are using ES-1.4.1.

Thanks.

On Thursday, April 2, 2015 at 3:02:31 AM UTC+5:30, Mark Walkom wrote:

It's not recommended to alter the .in.sh, this file can change between
releases and your alterations can either be lost or can cause other
problems.

I'd suggest you do what Gavin mentioned.

On 1 April 2015 at 18:08, Anil Karaka <anilk...@gmail.com <javascript:>>
wrote:

Thanks,

I just kept those GC logging options outside the if block.. So that I
don't have to set that variable, and enabled GC logging by default.

this is in the file /usr/share/elasticsearch/bin/elasticsearch.in.sh

GC logging options

if [ "x$ES_USE_GC_LOGGING" != "x" ]; then
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"
fi

Enable gc logging by default

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"

On Saturday, March 14, 2015 at 7:18:29 AM UTC+5:30, Gavin Seng wrote:

Anil, you can try this:

/etc/sysconfig/elasticsearch

export ES_USE_GC_LOGGING=1
ES_JAVA_OPTS="-verbose:gc -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
-Xloggc:"

Gavin

On Wednesday, March 4, 2015 at 9:22:50 AM UTC-5, Anil Karaka wrote:

Can you please tell me what exactly do I need to add in
/etc/default/elasticsearch?

What should I set ES_USE_GC_LOGGING to?

I added below line in my dev ES node, and I didn't see the gc.log file
in my log folder.. I want to be sure what to do before changing anything in
my production cluster..

ES_USE_GC_LOGGING=true

And what are the GC logging options in my elasticsearch.yml do? where
do those logs appear?

Thank you.

On Wednesday, March 4, 2015 at 3:11:26 AM UTC+5:30, Mark Walkom wrote:

You can set JAVA_OPTS in /etc/default/elasticsearch or
/etc/sysconfig/elasticsearch u9nder ES_JAVA_OPTS

On 3 March 2015 at 23:11, Anil Karaka anilk...@gmail.com wrote:

Some of the nodes are leaving the cluster from time to time..

I un commented the GC Logging options in elasticsearch.yml file as
shown and restarted my node.

################################## GC Logging
################################

monitor.jvm.gc.young.warn: 1000ms
monitor.jvm.gc.young.info: 700ms
monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s
monitor.jvm.gc.old.info: 5s
monitor.jvm.gc.old.debug: 2s

yet I'm not able to see any of the GC logging..

I came across this blog post http://tech.taskrabbit.com/
blog/2014/07/18/elasticsearch-in-production/

`ElasticSearch will note in its normal logs if any GC is happening
which effects a slow query, but we wanted more detail. Another logging
option ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM
variables:

from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi

`
Where do I set the ES_USE_GC_LOGGING option in my node..
so that I can log GC events..

currently there is no gc.log file in my logs folder..
just normal cluster logs and slow index/search logs..

Thanks..

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f807b943-6aad-4923-88db-d02cdb72595f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f807b943-6aad-4923-88db-d02cdb72595f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/22facccb-60fe-4080-87a1-2a5eaa78dd71%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

We don't recommend G1GC as it can lead to data loss and corruption in the
underlying lucene engine.

On 2 April 2015 at 16:49, Anil Karaka anilkaraka@gmail.com wrote:

Ok, we also had to edit that file to change the default garbage collector
options and are now using G1GC. So I thought I will just change the if
condition to enable gc logs..

Any way, this is not the main problem, I'm trying to do all these things
to debug the main issue of my nodes leaving the cluster randomly.
https://groups.google.com/forum/#!topic/elasticsearch/E-aGhovVTPI here
are more details. Any help would be appreciated.

Yesterday night I increased the zen.ping.timeout from 3s to 10s of all my
machines and restarted them one by one, and yet to see if its any
improvement.
Will updating elasticsearch in my nodes fix my issues? Currently my nodes
are using ES-1.4.1.

Thanks.

On Thursday, April 2, 2015 at 3:02:31 AM UTC+5:30, Mark Walkom wrote:

It's not recommended to alter the .in.sh, this file can change between
releases and your alterations can either be lost or can cause other
problems.

I'd suggest you do what Gavin mentioned.

On 1 April 2015 at 18:08, Anil Karaka anilk...@gmail.com wrote:

Thanks,

I just kept those GC logging options outside the if block.. So that I
don't have to set that variable, and enabled GC logging by default.

this is in the file /usr/share/elasticsearch/bin/elasticsearch.in.sh

GC logging options

if [ "x$ES_USE_GC_LOGGING" != "x" ]; then
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"
fi

Enable gc logging by default

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"

On Saturday, March 14, 2015 at 7:18:29 AM UTC+5:30, Gavin Seng wrote:

Anil, you can try this:

/etc/sysconfig/elasticsearch

export ES_USE_GC_LOGGING=1
ES_JAVA_OPTS="-verbose:gc -XX:+PrintGCDateStamps
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=10 -XX:GCLogFileSize=10M
-Xloggc:"

Gavin

On Wednesday, March 4, 2015 at 9:22:50 AM UTC-5, Anil Karaka wrote:

Can you please tell me what exactly do I need to add in
/etc/default/elasticsearch?

What should I set ES_USE_GC_LOGGING to?

I added below line in my dev ES node, and I didn't see the gc.log file
in my log folder.. I want to be sure what to do before changing anything in
my production cluster..

ES_USE_GC_LOGGING=true

And what are the GC logging options in my elasticsearch.yml do? where
do those logs appear?

Thank you.

On Wednesday, March 4, 2015 at 3:11:26 AM UTC+5:30, Mark Walkom wrote:

You can set JAVA_OPTS in /etc/default/elasticsearch or
/etc/sysconfig/elasticsearch u9nder ES_JAVA_OPTS

On 3 March 2015 at 23:11, Anil Karaka anilk...@gmail.com wrote:

Some of the nodes are leaving the cluster from time to time..

I un commented the GC Logging options in elasticsearch.yml file as
shown and restarted my node.

################################## GC Logging
################################

monitor.jvm.gc.young.warn: 1000ms
monitor.jvm.gc.young.info: 700ms
monitor.jvm.gc.young.debug: 400ms

monitor.jvm.gc.old.warn: 10s
monitor.jvm.gc.old.info: 5s
monitor.jvm.gc.old.debug: 2s

yet I'm not able to see any of the GC logging..

I came across this blog post http://tech.taskrabbit.com/blo
g/2014/07/18/elasticsearch-in-production/

`ElasticSearch will note in its normal logs if any GC is happening
which effects a slow query, but we wanted more detail. Another logging
option ElasticSearch has is ES_USE_GC_LOGGING which sets some JVM
variables:

from elasticsearch.in.sh.erbif [ "x$ES_USE_GC_LOGGING" != "x" ]; then

JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCDetails"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCTimeStamps"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintClassHistogram"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintTenuringDistribution"
JAVA_OPTS="$JAVA_OPTS -XX:+PrintGCApplicationStoppedTime"
JAVA_OPTS="$JAVA_OPTS -Xloggc:/var/log/elasticsearch/gc.log"fi

`
Where do I set the ES_USE_GC_LOGGING option in my node..
so that I can log GC events..

currently there is no gc.log file in my logs folder..
just normal cluster logs and slow index/search logs..

Thanks..

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b6
4-4883-b5d3-bd0f9e1b7312%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/8cc75569-1b64-4883-b5d3-bd0f9e1b7312%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/f807b943-6aad-4923-88db-d02cdb72595f%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/f807b943-6aad-4923-88db-d02cdb72595f%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/22facccb-60fe-4080-87a1-2a5eaa78dd71%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/22facccb-60fe-4080-87a1-2a5eaa78dd71%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-73FAkedn60cfaj2hu5t%2BFF1saJMeqUQ%2B6Owg8TLFNqg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.