After bulk indexing with refresh_interval disabled, now at 100% CPU usage for > 24 hours

Elastic Stack Elasticsearch
1

Hello,

We have a brand-new ES 1.0.1 cluster of 3 m2.xlarge machines, we set index.refresh_interval to -1, index.number_of_replicas to 0, index.number_of_shards to 10 and indexed about half a million documents in about 2000 indexes, this completed successfully in about 10 hours.

However, after the bulk indexing completed, I set index.refresh_interval to 5, and there is 100% CPU usage in 1 out of the 2 CPUs on all the 3 nodes, and it has been more than 24 hours and it is still at 100% CPU (1 out of 2 CPUs). Is this normal and expected? (Note that the cluster status is green)

From /_nodes/hot_threads, I can see that org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher.run(InternalIndexShard.java:914) is what is taking up the CPU.

Any advice on the same is welcome.

Thank you.

Regards,
Swaroop

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1371391394171884%40web12m.yandex.ru.
For more options, visit https://groups.google.com/d/optout.

2

For posterity note, the problem was solved by specifying {"index.refresh_interval": "5s"} - note the "s" : by specifying just "5", ES assumes 5 milliseconds!

Regards,
Swaroop

07.03.2014, 11:28, "Swaroop CH" swaroopch@yandex.com:

Hello,

We have a brand-new ES 1.0.1 cluster of 3 m2.xlarge machines, we set index.refresh_interval to -1, index.number_of_replicas to 0, index.number_of_shards to 10 and indexed about half a million documents in about 2000 indexes, this completed successfully in about 10 hours.

However, after the bulk indexing completed, I set index.refresh_interval to 5, and there is 100% CPU usage in 1 out of the 2 CPUs on all the 3 nodes, and it has been more than 24 hours and it is still at 100% CPU (1 out of 2 CPUs). Is this normal and expected? (Note that the cluster status is green)

From /_nodes/hot_threads, I can see that org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher.run(InternalIndexShard.java:914) is what is taking up the CPU.

Any advice on the same is welcome.

Thank you.

Regards,
Swaroop

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1371391394171884%40web12m.yandex.ru.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1411101394172329%40web12m.yandex.ru.
For more options, visit https://groups.google.com/d/optout.

3

I'm sometimes doing the same error with JVM memory parameters.

-Xmx512 -Xms512 definitely does not give enough memory to elasticsearch! :slight_smile:

--
David :wink:
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

Le 7 mars 2014 à 07:05, Swaroop CH swaroopch@yandex.com a écrit :

For posterity note, the problem was solved by specifying {"index.refresh_interval": "5s"} - note the "s" : by specifying just "5", ES assumes 5 milliseconds!

Regards,
Swaroop

07.03.2014, 11:28, "Swaroop CH" swaroopch@yandex.com:

Hello,

We have a brand-new ES 1.0.1 cluster of 3 m2.xlarge machines, we set index.refresh_interval to -1, index.number_of_replicas to 0, index.number_of_shards to 10 and indexed about half a million documents in about 2000 indexes, this completed successfully in about 10 hours.

However, after the bulk indexing completed, I set index.refresh_interval to 5, and there is 100% CPU usage in 1 out of the 2 CPUs on all the 3 nodes, and it has been more than 24 hours and it is still at 100% CPU (1 out of 2 CPUs). Is this normal and expected? (Note that the cluster status is green)

From /_nodes/hot_threads, I can see that org.elasticsearch.index.shard.service.InternalIndexShard$EngineRefresher.run(InternalIndexShard.java:914) is what is taking up the CPU.

Any advice on the same is welcome.

Thank you.

Regards,
Swaroop

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1371391394171884%40web12m.yandex.ru.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1411101394172329%40web12m.yandex.ru.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/012D35F8-C790-4E57-BBF8-AAABD82C73F2%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.