I've seen a few cases of "everything" slow down. Like this:
[image: Elasticsearch cluster eqiad es_query_time]
I mean, we don't have the most even service times at the moment, but woah!
I don't see anything in Elasticsearch's primary logs. The slow logs
abound. They show that things that should have taken a handful of
milliseconds spike to ~8 seconds. Whatever causes it just stops after a
minute or so and we get back to normal service times.
It looks like most of the disruption was mostly to one index but the extra
load slowed down other unrelated requests. I think. I've tried to tie
this to other things. It doesn't look like it corresponds to a merge or a
recovering shard or anything.
Any ideas what this is? Any ideas of what to turn up in the log file to
find out more?
The two candidates are merges and full GCs. Make sure you are monitoring
both of those.
On Wed, Feb 5, 2014 at 10:28 AM, Nikolas Everett nik9000@gmail.com wrote:
I've seen a few cases of "everything" slow down. Like this:
[image: Elasticsearch cluster eqiad es_query_time]
I mean, we don't have the most even service times at the moment, but
woah! I don't see anything in Elasticsearch's primary logs. The slow logs
abound. They show that things that should have taken a handful of
milliseconds spike to ~8 seconds. Whatever causes it just stops after a
minute or so and we get back to normal service times.
It looks like most of the disruption was mostly to one index but the extra
load slowed down other unrelated requests. I think. I've tried to tie
this to other things. It doesn't look like it corresponds to a merge or a
recovering shard or anything.
Any ideas what this is? Any ideas of what to turn up in the log file to
find out more?
Those were the first two I looked for and neither lined up. Also, my image
seems to have scrolled. I didn't really think about that part when I
pasted it into the email ;(
The two candidates are merges and full GCs. Make sure you are monitoring
both of those.
On Wed, Feb 5, 2014 at 10:28 AM, Nikolas Everett nik9000@gmail.comwrote:
I've seen a few cases of "everything" slow down. Like this:
[image: Elasticsearch cluster eqiad es_query_time]
I mean, we don't have the most even service times at the moment, but
woah! I don't see anything in Elasticsearch's primary logs. The slow logs
abound. They show that things that should have taken a handful of
milliseconds spike to ~8 seconds. Whatever causes it just stops after a
minute or so and we get back to normal service times.
It looks like most of the disruption was mostly to one index but the
extra load slowed down other unrelated requests. I think. I've tried to
tie this to other things. It doesn't look like it corresponds to a merge
or a recovering shard or anything.
Any ideas what this is? Any ideas of what to turn up in the log file to
find out more?
Take a look at the ElasticHQ, kopf and bigdesk plugins, they will provide
some more stats but also frame them relative to other operations that are
happening on the cluster/node.
Those were the first two I looked for and neither lined up. Also, my
image seems to have scrolled. I didn't really think about that part when I
pasted it into the email ;(
The two candidates are merges and full GCs. Make sure you are monitoring
both of those.
On Wed, Feb 5, 2014 at 10:28 AM, Nikolas Everett nik9000@gmail.comwrote:
I've seen a few cases of "everything" slow down. Like this:
[image: Elasticsearch cluster eqiad es_query_time]
I mean, we don't have the most even service times at the moment, but
woah! I don't see anything in Elasticsearch's primary logs. The slow logs
abound. They show that things that should have taken a handful of
milliseconds spike to ~8 seconds. Whatever causes it just stops after a
minute or so and we get back to normal service times.
It looks like most of the disruption was mostly to one index but the
extra load slowed down other unrelated requests. I think. I've tried to
tie this to other things. It doesn't look like it corresponds to a merge
or a recovering shard or anything.
Any ideas what this is? Any ideas of what to turn up in the log file to
find out more?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.