# Interesting CPU load (without actually traffic load) **URL:** \ **Category:** Elasticsearch\ **Created:** [March 8, 2012, 10:31pm UTC](https://discuss.elastic.co/t/interesting-cpu-load-without-actually-traffic-load/6955 "2012-03-08T22:31:07Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![jacque74](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jacque74](https://discuss.elastic.co/u/jacque74)\ **Post date:** [March 8, 2012, 10:31pm UTC](https://discuss.elastic.co/t/interesting-cpu-load-without-actually-traffic-load/6955/1 "2012-03-08T22:31:07Z")
Hello, every ones in a while we get into a state where one of our servers reports high USER and System CPU, as indicated on this graph: [http://bit.ly/yAQyPe](http://bit.ly/yAQyPe) As you can tell, the rest of the cluster is pretty much idle, while img699 is continuously hot with CPU top - 14:28:46 up 737 days, 18:21, 1 user, load average: 12.03, 10.37, 10.25 Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie Cpu(s): 34.9%us, 33.9%sy, 0.0%ni, 18.8%id, 11.7%wa, 0.2%hi, 0.5%si, 0.0%st Mem: 16472372k total, 16387968k used, 84404k free, 5952k buffers Swap: 9775544k total, 5632k used, 9769912k free, 6111504k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 8625 root 20 0 9113m 8.4g 10m S 513.6 53.6 9021:31 java Please see attached Jstack: [http://pastebin.com/HFUAn6ra](http://pastebin.com/HFUAn6ra) I am not really sure what its doing, all of the health status indicators are idle, there is no merge or flush in progress, and left alone, this server will be hot for days. The only way to resolve this is to restart the process. Shay, please let me know what you think. -Jack ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 9, 2012, 12:04am UTC](https://discuss.elastic.co/t/interesting-cpu-load-without-actually-traffic-load/6955/2 "2012-03-09T00:04:45Z")
Thanks for the stack trace. From what I can see, there are several on going stats requests on that node, and I can see when this might happen if a shard is being closed or was closed while they were executing (I can't see from the stack trace if thats the case). I fixed it here: [Closing a shard can cause a search/stats request that tries to acquire a handle to search while its closing to spin · Issue #1772 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1772). Otherwise, the only other thing that I can think is that the networking lib is causing it (there were some bugs related to that in older versions, though they have been fixed and I its not evident that tis happening from the stack trace). On Friday, March 9, 2012 at 12:31 AM, Jack Levin wrote: > Hello, every ones in a while we get into a state where one of our > servers reports high USER and System CPU, as indicated on this graph: > > [http://bit.ly/yAQyPe](http://bit.ly/yAQyPe) > > As you can tell, the rest of the cluster is pretty much idle, while > img699 is continuously hot with CPU > > top - 14:28:46 up 737 days, 18:21, 1 user, load average: 12.03, > 10.37, 10.25 > Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie > Cpu(s): 34.9%us, 33.9%sy, 0.0%ni, 18.8%id, 11.7%wa, 0.2%hi, > 0.5%si, 0.0%st > Mem: 16472372k total, 16387968k used, 84404k free, 5952k > buffers > Swap: 9775544k total, 5632k used, 9769912k free, 6111504k > cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 8625 root 20 0 9113m 8.4g 10m S 513.6 53.6 9021:31 java > > Please see attached Jstack: > > [Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode):"At - Pastebin.com](http://pastebin.com/HFUAn6ra) > > I am not really sure what its doing, all of the health status > indicators are idle, there is no merge or flush in progress, and left > alone, this server will be hot for days. The only way to resolve this > is to restart the process. > > Shay, please let me know what you think. > > -Jack ---
**Author:** ![jacque74](https://avatars.discourse-cdn.com/v4/letter/j/4af34b/32.png) [@jacque74](https://discuss.elastic.co/u/jacque74)\ **Post date:** [March 10, 2012, 2:37am UTC](https://discuss.elastic.co/t/interesting-cpu-load-without-actually-traffic-load/6955/3 "2012-03-10T02:37:12Z")
Shay, is there a stable version of ES I can get with this Fix? Otherwise where can I get it? -jack On Mar 8, 4:04 pm, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > Thanks for the stack trace. From what I can see, there are several on going stats requests on that node, and I can see when this might happen if a shard is being closed or was closed while they were executing (I can't see from the stack trace if thats the case). I fixed it here:[Closing a shard can cause a search/stats request that tries to acquire a handle to search while its closing to spin · Issue #1772 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1772). Otherwise, the only other thing that I can think is that the networking lib is causing it (there were some bugs related to that in older versions, though they have been fixed and I its not evident that tis happening from the stack trace). > > On Friday, March 9, 2012 at 12:31 AM, Jack Levin wrote: > > > Hello, every ones in a while we get into a state where one of our > > servers reports high USER and System CPU, as indicated on this graph: > > > [http://bit.ly/yAQyPe](http://bit.ly/yAQyPe) > > > As you can tell, the rest of the cluster is pretty much idle, while > > img699 is continuously hot with CPU > > > top - 14:28:46 up 737 days, 18:21, 1 user, load average: 12.03, > > 10.37, 10.25 > > Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie > > Cpu(s): 34.9%us, 33.9%sy, 0.0%ni, 18.8%id, 11.7%wa, 0.2%hi, > > 0.5%si, 0.0%st > > Mem: 16472372k total, 16387968k used, 84404k free, 5952k > > buffers > > Swap: 9775544k total, 5632k used, 9769912k free, 6111504k > > cached > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > > COMMAND > > 8625 root 20 0 9113m 8.4g 10m S 513.6 53.6 9021:31 java > > > Please see attached Jstack: > > > [Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode):"At - Pastebin.com](http://pastebin.com/HFUAn6ra) > > > I am not really sure what its doing, all of the health status > > indicators are idle, there is no merge or flush in progress, and left > > alone, this server will be hot for days. The only way to resolve this > > is to restart the process. > > > Shay, please let me know what you think. > > > -Jack ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 10, 2012, 6:48pm UTC](https://discuss.elastic.co/t/interesting-cpu-load-without-actually-traffic-load/6955/4 "2012-03-10T18:48:27Z")
The fix has been applied to both master and 0.19 branch. It will be part of 0.19.1 release (released either this week or the next). You can easily build a version yourself to test it, just clone / download the 0.19 branch from github, and run "mvn package -DskipTests", the distribution files will be under target/release. On Saturday, March 10, 2012 at 4:37 AM, Jack Levin wrote: > Shay, is there a stable version of ES I can get with this Fix? > Otherwise where can I get it? > > -jack > > On Mar 8, 4:04 pm, Shay Banon \<[kim...@gmail.com](mailto:kim...@gmail.com) ([http://gmail.com](http://gmail.com))\> wrote: > > > Thanks for the stack trace. From what I can see, there are several on going stats requests on that node, and I can see when this might happen if a shard is being closed or was closed while they were executing (I can't see from the stack trace if thats the case). I fixed it here:[Closing a shard can cause a search/stats request that tries to acquire a handle to search while its closing to spin · Issue #1772 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1772). Otherwise, the only other thing that I can think is that the networking lib is causing it (there were some bugs related to that in older versions, though they have been fixed and I its not evident that tis happening from the stack trace). > > > > On Friday, March 9, 2012 at 12:31 AM, Jack Levin wrote: > > > > > Hello, every ones in a while we get into a state where one of our > > > servers reports high USER and System CPU, as indicated on this graph: > > > > > [http://bit.ly/yAQyPe](http://bit.ly/yAQyPe) > > > > > As you can tell, the rest of the cluster is pretty much idle, while > > > img699 is continuously hot with CPU > > > > > top - 14:28:46 up 737 days, 18:21, 1 user, load average: 12.03, > > > 10.37, 10.25 > > > Tasks: 125 total, 1 running, 124 sleeping, 0 stopped, 0 zombie > > > Cpu(s): 34.9%us, 33.9%sy, 0.0%ni, 18.8%id, 11.7%wa, 0.2%hi, > > > 0.5%si, 0.0%st > > > Mem: 16472372k total, 16387968k used, 84404k free, 5952k > > > buffers > > > Swap: 9775544k total, 5632k used, 9769912k free, 6111504k > > > cached > > > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > > > COMMAND > > > 8625 root 20 0 9113m 8.4g 10m S 513.6 53.6 9021:31 java > > > > > Please see attached Jstack: > > > > > [Full thread dump Java HotSpot(TM) 64-Bit Server VM (19.1-b02 mixed mode):"At - Pastebin.com](http://pastebin.com/HFUAn6ra) > > > > > I am not really sure what its doing, all of the health status > > > indicators are idle, there is no merge or flush in progress, and left > > > alone, this server will be hot for days. The only way to resolve this > > > is to restart the process. > > > > > Shay, please let me know what you think. > > > > > -Jack ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:36am UTC](https://discuss.elastic.co/t/interesting-cpu-load-without-actually-traffic-load/6955/5 "2017-07-06T03:36:24Z")