# Sudden 100% CPU usage by ElasticSearch **URL:** \ **Category:** Elasticsearch\ **Created:** [March 27, 2014, 8:00pm UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657 "2014-03-27T20:00:35Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Jos\_Kraaijeveld](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jos_kraaijeveld/32/1668_2.png) [@Jos\_Kraaijeveld](https://discuss.elastic.co/u/Jos_Kraaijeveld)\ **Post date:** [March 27, 2014, 8:00pm UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657/1 "2014-03-27T20:00:35Z")
At seemingly random intervals, the ElasticSearch Java process starts hogging all CPU on my machine. This is in a two-node cluster where one node is gathering data from other sources and continuously updating the documents. Documents time out when they haven't been updated for a while. The documents get replicated to the other node in a single shard setup. This happened without any querying going on aside from very trivial health checks. The node actually performing the updating starts using max CPU until I restart ElasticSearch, but I can't figure out why. To show the effect, here's a Ganglia graph: [https://lh4.googleusercontent.com/-P\_81cbz2uQU/UzSCGjuJzNI/AAAAAAAAAB8/2zSevfkpN3c/s1600/es\_cpu.png](https://lh4.googleusercontent.com/-P_81cbz2uQU/UzSCGjuJzNI/AAAAAAAAAB8/2zSevfkpN3c/s1600/es_cpu.png) This is a 24GB machine with 24 cores, running ElasticSearch 1.0.1 on OpenJDK 7. I took a long snapshot of hot\_threads when it was happening, it's available over here: [https://gist.github.com/Kaidence/2b95c207f4e6a79841c5](https://gist.github.com/Kaidence/2b95c207f4e6a79841c5). I was wondering whether someone had seen this before or had any clue why this is happening. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e7ecc7db-b310-448f-8d09-c3985e6c3564%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e7ecc7db-b310-448f-8d09-c3985e6c3564%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jos\_Kraaijeveld](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jos_kraaijeveld/32/1668_2.png) [@Jos\_Kraaijeveld](https://discuss.elastic.co/u/Jos_Kraaijeveld)\ **Post date:** [March 27, 2014, 8:02pm UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657/2 "2014-03-27T20:02:20Z")
Apologies, the gist link wasn't working properly: [ElasticSearch eating all CPU · GitHub](https://gist.github.com/Kaidence/2b95c207f4e6a79841c5)[https://gist.github.com/Kaidence/2b95c207f4e6a79841c5](https://gist.github.com/Kaidence/2b95c207f4e6a79841c5) On Thursday, March 27, 2014 1:00:35 PM UTC-7, Jos Kraaijeveld wrote: > At seemingly random intervals, the Elasticsearch Java process starts > hogging all CPU on my machine. This is in a two-node cluster where one node > is gathering data from other sources and continuously updating the > documents. Documents time out when they haven't been updated for a while. > The documents get replicated to the other node in a single shard setup. > This happened without any querying going on aside from very trivial health > checks. > > The node actually performing the updating starts using max CPU until I > restart Elasticsearch, but I can't figure out why. To show the effect, > here's a Ganglia graph: > > [https://lh4.googleusercontent.com/-P\_81cbz2uQU/UzSCGjuJzNI/AAAAAAAAAB8/2zSevfkpN3c/s1600/es\_cpu.png](https://lh4.googleusercontent.com/-P_81cbz2uQU/UzSCGjuJzNI/AAAAAAAAAB8/2zSevfkpN3c/s1600/es_cpu.png) > > This is a 24GB machine with 24 cores, running Elasticsearch 1.0.1 on > OpenJDK 7. I took a long snapshot of hot\_threads when it was happening, > it's available over here: > [ElasticSearch eating all CPU · GitHub](https://gist.github.com/Kaidence/2b95c207f4e6a79841c5). > > I was wondering whether someone had seen this before or had any clue why > this is happening. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/66851a83-f5ca-4a52-becc-37a8b5052ef1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/66851a83-f5ca-4a52-becc-37a8b5052ef1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [March 27, 2014, 8:09pm UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657/3 "2014-03-27T20:09:01Z")
Threads look like four concurrent scans but I can't be sure what they are up to. Nik On Thu, Mar 27, 2014 at 4:02 PM, Jos Kraaijeveld [mail@kaidence.org](mailto:mail@kaidence.org) wrote: > Apologies, the gist link wasn't working properly: > [ElasticSearch eating all CPU · GitHub](https://gist.github.com/Kaidence/2b95c207f4e6a79841c5)[https://gist.github.com/Kaidence/2b95c207f4e6a79841c5](https://gist.github.com/Kaidence/2b95c207f4e6a79841c5) > > On Thursday, March 27, 2014 1:00:35 PM UTC-7, Jos Kraaijeveld wrote: > > > At seemingly random intervals, the Elasticsearch Java process starts > > hogging all CPU on my machine. This is in a two-node cluster where one node > > is gathering data from other sources and continuously updating the > > documents. Documents time out when they haven't been updated for a while. > > The documents get replicated to the other node in a single shard setup. > > This happened without any querying going on aside from very trivial health > > checks. > > > > The node actually performing the updating starts using max CPU until I > > restart Elasticsearch, but I can't figure out why. To show the effect, > > here's a Ganglia graph: > > > > [https://lh4.googleusercontent.com/-P\_81cbz2uQU/UzSCGjuJzNI/AAAAAAAAAB8/2zSevfkpN3c/s1600/es\_cpu.png](https://lh4.googleusercontent.com/-P_81cbz2uQU/UzSCGjuJzNI/AAAAAAAAAB8/2zSevfkpN3c/s1600/es_cpu.png) > > > > This is a 24GB machine with 24 cores, running Elasticsearch 1.0.1 on > > OpenJDK 7. I took a long snapshot of hot\_threads when it was happening, > > it's available over here: [https://gist.github.com/](https://gist.github.com/) > > Kaidence/2b95c207f4e6a79841c5. > > > > I was wondering whether someone had seen this before or had any clue why > > this is happening. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/66851a83-f5ca-4a52-becc-37a8b5052ef1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/66851a83-f5ca-4a52-becc-37a8b5052ef1%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/66851a83-f5ca-4a52-becc-37a8b5052ef1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/66851a83-f5ca-4a52-becc-37a8b5052ef1%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0fYaHBOpg9qBBqhYHf5eovJRkOpaT7L%3DkCHYO1mGW83Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAPmjWd0fYaHBOpg9qBBqhYHf5eovJRkOpaT7L%3DkCHYO1mGW83Q%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 27, 2014, 8:46pm UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657/4 "2014-03-27T20:46:02Z")
How many documents do you scan by your queries, and how long is the lifetime of a scroll query? Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHkttZixretx9z%2BABG8M6NQX%2BRTO2Cues0P2cxH1w50Cg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoHkttZixretx9z%2BABG8M6NQX%2BRTO2Cues0P2cxH1w50Cg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jos\_Kraaijeveld](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jos_kraaijeveld/32/1668_2.png) [@Jos\_Kraaijeveld](https://discuss.elastic.co/u/Jos_Kraaijeveld)\ **Post date:** [March 27, 2014, 8:47pm UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657/5 "2014-03-27T20:47:54Z")
There are at most 15k documents alive, I scan them 50 at a time. Each scroll query lives for 3 minutes. On Thursday, March 27, 2014 1:46:02 PM UTC-7, Jörg Prante wrote: > How many documents do you scan by your queries, and how long is the > lifetime of a scroll query? > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [March 27, 2014, 9:08pm UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657/6 "2014-03-27T21:08:29Z")
You should reduce the 3 minutes to the absolute minimum (maybe you use the hits within a few seconds). Jörg On Thu, Mar 27, 2014 at 9:47 PM, Jos Kraaijeveld [mail@kaidence.org](mailto:mail@kaidence.org) wrote: > There are at most 15k documents alive, I scan them 50 at a time. Each > scroll query lives for 3 minutes. > > On Thursday, March 27, 2014 1:46:02 PM UTC-7, Jörg Prante wrote: > > > How many documents do you scan by your queries, and how long is the > > lifetime of a scroll query? > > > > Jörg > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFc\_8Vm3D762xfy7DR3ch7tm11dRxXEnq6LSy4e2Ox4eA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFc_8Vm3D762xfy7DR3ch7tm11dRxXEnq6LSy4e2Ox4eA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jos\_Kraaijeveld](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jos_kraaijeveld/32/1668_2.png) [@Jos\_Kraaijeveld](https://discuss.elastic.co/u/Jos_Kraaijeveld)\ **Post date:** [March 28, 2014, 6:26pm UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657/7 "2014-03-28T18:26:01Z")
In the meantime I've upgraded to Elasticsearch 1.1.0 and reduced the scroll time to a few seconds like you recommended. I just had the same thing happen again with a very similar hot\_threads response. This is preventing me from properly running ES in production and I'm running out of ideas - any help is appreciated. On Thursday, March 27, 2014 2:08:29 PM UTC-7, Jörg Prante wrote: > You should reduce the 3 minutes to the absolute minimum (maybe you use the > hits within a few seconds). > > Jörg > > On Thu, Mar 27, 2014 at 9:47 PM, Jos Kraaijeveld \<[ma...@kaidence.org](mailto:ma...@kaidence.org)\ > > > wrote: > > > There are at most 15k documents alive, I scan them 50 at a time. Each > > scroll query lives for 3 minutes. > > > > On Thursday, March 27, 2014 1:46:02 PM UTC-7, Jörg Prante wrote: > > > > > How many documents do you scan by your queries, and how long is the > > > lifetime of a scroll query? > > > > > > Jörg > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/299fb2f1-73f1-494a-98f7-a7e9f6b1d3bb%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/299fb2f1-73f1-494a-98f7-a7e9f6b1d3bb%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Jos\_Kraaijeveld](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jos_kraaijeveld/32/1668_2.png) [@Jos\_Kraaijeveld](https://discuss.elastic.co/u/Jos_Kraaijeveld)\ **Post date:** [April 3, 2014, 6:21pm UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657/8 "2014-04-03T18:21:25Z")
I am still encountering this issue. Can someone point me in a direction of where to debug? Is this is an issue of Lucene or am I using scan queries incorrectly? On Friday, March 28, 2014 11:26:01 AM UTC-7, Jos Kraaijeveld wrote: > In the meantime I've upgraded to Elasticsearch 1.1.0 and reduced the > scroll time to a few seconds like you recommended. I just had the same > thing happen again with a very similar hot\_threads response. This is > preventing me from properly running ES in production and I'm running out of > ideas - any help is appreciated. > > On Thursday, March 27, 2014 2:08:29 PM UTC-7, Jörg Prante wrote: > > > You should reduce the 3 minutes to the absolute minimum (maybe you use > > the hits within a few seconds). > > > > Jörg > > > > On Thu, Mar 27, 2014 at 9:47 PM, Jos Kraaijeveld [ma...@kaidence.org](mailto:ma...@kaidence.org)wrote: > > > > > There are at most 15k documents alive, I scan them 50 at a time. Each > > > scroll query lives for 3 minutes. > > > > > > On Thursday, March 27, 2014 1:46:02 PM UTC-7, Jörg Prante wrote: > > > > > > > How many documents do you scan by your queries, and how long is the > > > > lifetime of a scroll query? > > > > > > > > Jörg > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/5478568b-38cd-4039-89ad-f088aa33a088%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5d7e89dd-b251-49a7-8505-208f68d1f333%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/5d7e89dd-b251-49a7-8505-208f68d1f333%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:38am UTC](https://discuss.elastic.co/t/sudden-100-cpu-usage-by-elasticsearch/16657/9 "2017-07-06T01:38:29Z")