Problems after upgrade to 0.90.7

Elastic Stack Elasticsearch
1

hi!

after upgrading cluster from 0.20.5 to 0.90.7 which went good , I'm
monitoring one issue which appeared: every day at exactly same time the CPU
load created by elasticsearch java process on all cluster nodes increases
from 2-3% to 90% for 15 minutes

it started exactly after upgrade

I checked periodic tasks, load from application, and other places:

  1. there are no cron jobs at that time
  2. the application activity is not high at that moment - the users's load
    is even lower at that time

so all points to the fact that this was caused after upgrade

any thoughts on how to act further?

thanks

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Just a wild guess, could it be gc? or could probably turn on the logging to
debug could give some hints.

/Jason

On Tue, Nov 26, 2013 at 11:18 PM, tetlika tetlika@gmail.com wrote:

hi!

after upgrading cluster from 0.20.5 to 0.90.7 which went good , I'm
monitoring one issue which appeared: every day at exactly same time the CPU
load created by elasticsearch java process on all cluster nodes increases
from 2-3% to 90% for 15 minutes

it started exactly after upgrade

I checked periodic tasks, load from application, and other places:

  1. there are no cron jobs at that time
  2. the application activity is not high at that moment - the users's load
    is even lower at that time

so all points to the fact that this was caused after upgrade

any thoughts on how to act further?

thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHO4itxZF3xLB8j73eNKQ2Jzu_B5dUY5yYyx8YRorasaRyu5HQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hey,

what time is it? Can you see if the load is actually caused by the java
process? If it is you can use the hot_threads API to investigate further,
in case this happens again.

See

--Alex

On Tue, Nov 26, 2013 at 9:46 PM, Jason Wee peichieh@gmail.com wrote:

Just a wild guess, could it be gc? or could probably turn on the logging
to debug could give some hints.

/Jason

On Tue, Nov 26, 2013 at 11:18 PM, tetlika tetlika@gmail.com wrote:

hi!

after upgrading cluster from 0.20.5 to 0.90.7 which went good , I'm
monitoring one issue which appeared: every day at exactly same time the CPU
load created by elasticsearch java process on all cluster nodes increases
from 2-3% to 90% for 15 minutes

it started exactly after upgrade

I checked periodic tasks, load from application, and other places:

  1. there are no cron jobs at that time
  2. the application activity is not high at that moment - the users's load
    is even lower at that time

so all points to the fact that this was caused after upgrade

any thoughts on how to act further?

thanks

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHO4itxZF3xLB8j73eNKQ2Jzu_B5dUY5yYyx8YRorasaRyu5HQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8sPW8fPCN6WQmLwqhg%2B9%3DXy9T6aeHmSS%2B-S4JoEg159g%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

we found the reason of the load spike - and it was not upgrade to 0.90.7

it was scheduled task which was hard to find :slight_smile:

thanks for helping

Середа, 27 листопада 2013 р. 12:43:07 UTC+2 користувач Alexander Reelsen
написав:

Hey,

what time is it? Can you see if the load is actually caused by the java
process? If it is you can use the hot_threads API to investigate further,
in case this happens again.

See
Elasticsearch Platform — Find real-time answers at scale | Elastic

--Alex

On Tue, Nov 26, 2013 at 9:46 PM, Jason Wee <peic...@gmail.com<javascript:>

wrote:

Just a wild guess, could it be gc? or could probably turn on the logging
to debug could give some hints.

/Jason

On Tue, Nov 26, 2013 at 11:18 PM, tetlika <tet...@gmail.com <javascript:>

wrote:

hi!

after upgrading cluster from 0.20.5 to 0.90.7 which went good , I'm
monitoring one issue which appeared: every day at exactly same time the CPU
load created by elasticsearch java process on all cluster nodes increases
from 2-3% to 90% for 15 minutes

it started exactly after upgrade

I checked periodic tasks, load from application, and other places:

  1. there are no cron jobs at that time
  2. the application activity is not high at that moment - the users's
    load is even lower at that time

so all points to the fact that this was caused after upgrade

any thoughts on how to act further?

thanks

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHO4itxZF3xLB8j73eNKQ2Jzu_B5dUY5yYyx8YRorasaRyu5HQ%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c34c8292-b21a-4325-b0ff-e14ee887c3bf%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.