Excessive number of "rescheduling interrupts"

Elastic Stack Elasticsearch
1

I noticed some slowness on one of my EC2 instances running Amazon
Linux today, and discovered that my install of Elastic Search is
chewing up hundreds of thousands of interrupts every second:

There is very little load on that instance, and just to make sure my
indexer wasn't at fault, I stopped it. Then I stopped Elastic Search,
and saw the number of interrupts as reported by vmstat drop to almost
zero.

I am running Elastic Search version 0.18.5, on a single node with no clustering.

Can anyone offer any suggestions for things I can look at or check?

Thanks,

-- Doug

--
http://twitter.com/dmuth

2

That's a crazy high number. Have you tried this on a different/multiple
servers? Different OS? Kernel? JVM?

Otis

Search Analytics - Cloud Monitoring Tools & Services | Sematext
Scalable Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Wednesday, July 4, 2012 4:22:30 PM UTC-4, Douglas Muth wrote:

I noticed some slowness on one of my EC2 instances running Amazon
Linux today, and discovered that my install of Elastic Search is
chewing up hundreds of thousands of interrupts every second:

http://i.imgur.com/O0P9r.png

There is very little load on that instance, and just to make sure my
indexer wasn't at fault, I stopped it. Then I stopped Elastic Search,
and saw the number of interrupts as reported by vmstat drop to almost
zero.

I am running Elastic Search version 0.18.5, on a single node with no
clustering.

Can anyone offer any suggestions for things I can look at or check?

Thanks,

-- Doug

--
http://twitter.com/dmuth

3

Leapsecond effect ?

David :wink:
Twitter : @dadoonet / @elasticsearchfr

Le 5 juil. 2012 à 07:16, Otis Gospodnetic otis.gospodnetic@gmail.com a écrit :

That's a crazy high number. Have you tried this on a different/multiple servers? Different OS? Kernel? JVM?

Otis

Search Analytics - Cloud Monitoring Tools & Services | Sematext
Scalable Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Wednesday, July 4, 2012 4:22:30 PM UTC-4, Douglas Muth wrote:
I noticed some slowness on one of my EC2 instances running Amazon
Linux today, and discovered that my install of Elastic Search is
chewing up hundreds of thousands of interrupts every second:

http://i.imgur.com/O0P9r.png

There is very little load on that instance, and just to make sure my
indexer wasn't at fault, I stopped it. Then I stopped Elastic Search,
and saw the number of interrupts as reported by vmstat drop to almost
zero.

I am running Elastic Search version 0.18.5, on a single node with no clustering.

Can anyone offer any suggestions for things I can look at or check?

Thanks,

-- Doug

--
http://twitter.com/dmuth

4

On Thu, Jul 5, 2012 at 1:54 AM, David Pilato david@pilato.fr wrote:

Leapsecond effect ?

Revisiting this old thread, I just wanted to update everyone with the
resolution. It WAS in fact a leap-second issue! I wasn't sure if
that was the culprit at first, since it was right around the time
Amazon Web Services had that EC2 outage, and I didn't know if it might
have been related to that or not. (we had issues with some of our
other instances)

That said, here was the fix:

/etc/init.d/ntp stop; date; date date +"%m%d%H%M%C%y.%S"; date;

CPU usage of Elastic Search, and the interrupt usage dropped to
pre-leap second levels instantly. :slight_smile:

-- Doug