Everything just slows down simetimes


(Nik Everett) #1

I've seen a few cases of "everything" slow down. Like this:
[image: Elasticsearch cluster eqiad es_query_time]
I mean, we don't have the most even service times at the moment, but woah!
I don't see anything in Elasticsearch's primary logs. The slow logs
abound. They show that things that should have taken a handful of
milliseconds spike to ~8 seconds. Whatever causes it just stops after a
minute or so and we get back to normal service times.

It looks like most of the disruption was mostly to one index but the extra
load slowed down other unrelated requests. I think. I've tried to tie
this to other things. It doesn't look like it corresponds to a merge or a
recovering shard or anything.

Any ideas what this is? Any ideas of what to turn up in the log file to
find out more?

Nik

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3iwAEOQnXOxDk3_SB7xPQOig0rDRVHSwoJ47syDvAOfQ%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Randall McRee) #2

The two candidates are merges and full GCs. Make sure you are monitoring
both of those.

On Wed, Feb 5, 2014 at 10:28 AM, Nikolas Everett nik9000@gmail.com wrote:

I've seen a few cases of "everything" slow down. Like this:
[image: Elasticsearch cluster eqiad es_query_time]
I mean, we don't have the most even service times at the moment, but
woah! I don't see anything in Elasticsearch's primary logs. The slow logs
abound. They show that things that should have taken a handful of
milliseconds spike to ~8 seconds. Whatever causes it just stops after a
minute or so and we get back to normal service times.

It looks like most of the disruption was mostly to one index but the extra
load slowed down other unrelated requests. I think. I've tried to tie
this to other things. It doesn't look like it corresponds to a merge or a
recovering shard or anything.

Any ideas what this is? Any ideas of what to turn up in the log file to
find out more?

Nik

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3iwAEOQnXOxDk3_SB7xPQOig0rDRVHSwoJ47syDvAOfQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAFjHw34U4T9bGQ1eRMzB1-oxj7AhhrR9dRbp1-yNZofFTy3Zgw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Nik Everett) #3

Those were the first two I looked for and neither lined up. Also, my image
seems to have scrolled. I didn't really think about that part when I
pasted it into the email ;(

Nik

On Wed, Feb 5, 2014 at 1:45 PM, Randall McRee randall.mcree@gmail.comwrote:

The two candidates are merges and full GCs. Make sure you are monitoring
both of those.

On Wed, Feb 5, 2014 at 10:28 AM, Nikolas Everett nik9000@gmail.comwrote:

I've seen a few cases of "everything" slow down. Like this:
[image: Elasticsearch cluster eqiad es_query_time]
I mean, we don't have the most even service times at the moment, but
woah! I don't see anything in Elasticsearch's primary logs. The slow logs
abound. They show that things that should have taken a handful of
milliseconds spike to ~8 seconds. Whatever causes it just stops after a
minute or so and we get back to normal service times.

It looks like most of the disruption was mostly to one index but the
extra load slowed down other unrelated requests. I think. I've tried to
tie this to other things. It doesn't look like it corresponds to a merge
or a recovering shard or anything.

Any ideas what this is? Any ideas of what to turn up in the log file to
find out more?

Nik

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3iwAEOQnXOxDk3_SB7xPQOig0rDRVHSwoJ47syDvAOfQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAFjHw34U4T9bGQ1eRMzB1-oxj7AhhrR9dRbp1-yNZofFTy3Zgw%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd16pXWuLoEOPqce5j8gX7dSctrxXW1n0rKKPM%3DfHPb3mA%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Mark Walkom) #4

Take a look at the ElasticHQ, kopf and bigdesk plugins, they will provide
some more stats but also frame them relative to other operations that are
happening on the cluster/node.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 6 February 2014 06:18, Nikolas Everett nik9000@gmail.com wrote:

Those were the first two I looked for and neither lined up. Also, my
image seems to have scrolled. I didn't really think about that part when I
pasted it into the email ;(

Nik

On Wed, Feb 5, 2014 at 1:45 PM, Randall McRee randall.mcree@gmail.comwrote:

The two candidates are merges and full GCs. Make sure you are monitoring
both of those.

On Wed, Feb 5, 2014 at 10:28 AM, Nikolas Everett nik9000@gmail.comwrote:

I've seen a few cases of "everything" slow down. Like this:
[image: Elasticsearch cluster eqiad es_query_time]
I mean, we don't have the most even service times at the moment, but
woah! I don't see anything in Elasticsearch's primary logs. The slow logs
abound. They show that things that should have taken a handful of
milliseconds spike to ~8 seconds. Whatever causes it just stops after a
minute or so and we get back to normal service times.

It looks like most of the disruption was mostly to one index but the
extra load slowed down other unrelated requests. I think. I've tried to
tie this to other things. It doesn't look like it corresponds to a merge
or a recovering shard or anything.

Any ideas what this is? Any ideas of what to turn up in the log file to
find out more?

Nik

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd3iwAEOQnXOxDk3_SB7xPQOig0rDRVHSwoJ47syDvAOfQ%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAFjHw34U4T9bGQ1eRMzB1-oxj7AhhrR9dRbp1-yNZofFTy3Zgw%40mail.gmail.com
.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPmjWd16pXWuLoEOPqce5j8gX7dSctrxXW1n0rKKPM%3DfHPb3mA%40mail.gmail.com
.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624ZArhW4VVQMM0areaUT-8-ZxUBr2CEv%3DHhqs-px%3D-36kw%40mail.gmail.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #5