Excessive number of "rescheduling interrupts"


(Douglas Muth) #1

I noticed some slowness on one of my EC2 instances running Amazon
Linux today, and discovered that my install of Elastic Search is
chewing up hundreds of thousands of interrupts every second:

There is very little load on that instance, and just to make sure my
indexer wasn't at fault, I stopped it. Then I stopped Elastic Search,
and saw the number of interrupts as reported by vmstat drop to almost
zero.

I am running Elastic Search version 0.18.5, on a single node with no clustering.

Can anyone offer any suggestions for things I can look at or check?

Thanks,

-- Doug

--
http://twitter.com/dmuth


(Otis Gospodnetić) #2

That's a crazy high number. Have you tried this on a different/multiple
servers? Different OS? Kernel? JVM?

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, July 4, 2012 4:22:30 PM UTC-4, Douglas Muth wrote:

I noticed some slowness on one of my EC2 instances running Amazon
Linux today, and discovered that my install of Elastic Search is
chewing up hundreds of thousands of interrupts every second:

http://i.imgur.com/O0P9r.png

There is very little load on that instance, and just to make sure my
indexer wasn't at fault, I stopped it. Then I stopped Elastic Search,
and saw the number of interrupts as reported by vmstat drop to almost
zero.

I am running Elastic Search version 0.18.5, on a single node with no
clustering.

Can anyone offer any suggestions for things I can look at or check?

Thanks,

-- Doug

--
http://twitter.com/dmuth


(David Pilato) #3

Leapsecond effect ?

David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 5 juil. 2012 à 07:16, Otis Gospodnetic otis.gospodnetic@gmail.com a écrit :

That's a crazy high number. Have you tried this on a different/multiple servers? Different OS? Kernel? JVM?

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, July 4, 2012 4:22:30 PM UTC-4, Douglas Muth wrote:
I noticed some slowness on one of my EC2 instances running Amazon
Linux today, and discovered that my install of Elastic Search is
chewing up hundreds of thousands of interrupts every second:

http://i.imgur.com/O0P9r.png

There is very little load on that instance, and just to make sure my
indexer wasn't at fault, I stopped it. Then I stopped Elastic Search,
and saw the number of interrupts as reported by vmstat drop to almost
zero.

I am running Elastic Search version 0.18.5, on a single node with no clustering.

Can anyone offer any suggestions for things I can look at or check?

Thanks,

-- Doug

--
http://twitter.com/dmuth


(Douglas Muth) #4

On Thu, Jul 5, 2012 at 1:54 AM, David Pilato david@pilato.fr wrote:

Leapsecond effect ?

Revisiting this old thread, I just wanted to update everyone with the
resolution. It WAS in fact a leap-second issue! I wasn't sure if
that was the culprit at first, since it was right around the time
Amazon Web Services had that EC2 outage, and I didn't know if it might
have been related to that or not. (we had issues with some of our
other instances)

That said, here was the fix:

/etc/init.d/ntp stop; date; date date +"%m%d%H%M%C%y.%S"; date;

CPU usage of Elastic Search, and the interrupt usage dropped to
pre-leap second levels instantly. :slight_smile:

-- Doug


(system) #5