# Very high CPU usage of ES in idle state **URL:** \ **Category:** Elasticsearch\ **Created:** [April 9, 2013, 10:07am UTC](https://discuss.elastic.co/t/very-high-cpu-usage-of-es-in-idle-state/11499 "2013-04-09T10:07:32Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Talha\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talha_khan/32/2394_2.png) [@Talha\_Khan](https://discuss.elastic.co/u/Talha_Khan)\ **Post date:** [April 9, 2013, 10:07am UTC](https://discuss.elastic.co/t/very-high-cpu-usage-of-es-in-idle-state/11499/1 "2013-04-09T10:07:32Z")
I have set up a 1 node cluster of ES using ES version 0.19.11 for storing some logs. After inserting about 5000 logs I now see that my CPU usage is rocketing above 100% constanltly even when no/very few search queries are reaching it. ES has become very unresponsive and some of my search queries randomly return the following exception. "error" : "EsRejectedExecutionException[rejected execution of [org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]", "status" : 503 By randomly i mean sometimes they succeed but mostly they fail. Here is my ES config: index.cache.field.expire: 20m index.cache.field.type: soft bootstrap.mlockall: true threadpool: search: type: fixed size: 20 queue\_size: 1000 index: type: fixed size: 16 bulk: type: fixed size: 5 I did some digging into the issue. The stats for node\_stats, hot\_threads api and jstack can be found in this gist > I am also attaching the results of bigdesk. The cluster health is green but looking at these results, it seems that ES is very low on heap memory. But i am unable to understand the constant high CPU usage even in the idle state. How can this issue be solved? Kindly give some context and explanation to any answer. Thanks -atk -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 9, 2013, 2:27pm UTC](https://discuss.elastic.co/t/very-high-cpu-usage-of-es-in-idle-state/11499/2 "2013-04-09T14:27:04Z")
Note, for your workload of around 100 qps, I have the impression you have selected small memory for heap, around 350MB (?), and your GC is around 10-20 seconds, but your RAM is total of 2 GB. This indicates you should consider increasing the heap. GC times are going through the roof and they give you the high CPU and unresponsiveness. On a machine of 2 GB RAM there is a higher risk of bootstrap.mlockall: true to starve your overall system processes because you will quickly force them into swapping/paging, or you run out of real memory. So consider removing this option to give the system enough room or upgrade to a machine with \> 4 GB RAM. Another option is adding more nodes of course. If you stay with 2 GB RAM I would also recommend to disable 64-bit server JVM with "java -d32 -client" in order to save some bits on the heap. 64-bit server JVM makes sense on machines where you can use virtual address space in the area of \> 4GB RAM. Jörg Am 09.04.13 12:07, schrieb Talha Khan: > I have set up a 1 node cluster of ES using ES version 0.19.11 for > storing some logs. After inserting about 5000 logs I now see that my > CPU usage is rocketing above 100% constanltly even when no/very few > search queries are reaching it. ES has become very unresponsive and > some of my search queries randomly return the following exception. > "error" : "EsRejectedExecutionException[rejected execution of > [org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]", > "status" : 503 > > By randomly i mean sometimes they succeed but mostly they fail. > > Here is my ES config: > > index.cache.field.expire: 20m > index.cache.field.type: soft > > bootstrap.mlockall: true > threadpool: > search: > type: fixed > size: 20 > queue\_size: 1000 > index: > type: fixed > size: 16 > bulk: > type: fixed > size: 5 > > I did some digging into the issue. The stats for node\_stats, > hot\_threads api and jstack can be found in this gist > > [Elasticsearch high cpu · GitHub](https://gist.github.com/ahmedtalhakhan/5343784) > > I am also attaching the results of bigdesk. The cluster health is > green but looking at these results, it seems that ES is very low on > heap memory. But i am unable to understand the constant high CPU usage > even in the idle state. How can this issue be solved? > > Kindly give some context and explanation to any answer. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Talha\_Khan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/talha_khan/32/2394_2.png) [@Talha\_Khan](https://discuss.elastic.co/u/Talha_Khan)\ **Post date:** [April 9, 2013, 6:44pm UTC](https://discuss.elastic.co/t/very-high-cpu-usage-of-es-in-idle-state/11499/3 "2013-04-09T18:44:22Z")
I will try out the recommendations that you suggested. Will post my results. On Tuesday, 9 April 2013 19:27:04 UTC+5, Jörg Prante wrote: > Note, for your workload of around 100 qps, I have the impression you > have selected small memory for heap, around 350MB (?), and your GC is > around 10-20 seconds, but your RAM is total of 2 GB. This indicates you > should consider increasing the heap. GC times are going through the roof > and they give you the high CPU and unresponsiveness. > > On a machine of 2 GB RAM there is a higher risk of bootstrap.mlockall: > true to starve your overall system processes because you will quickly > force them into swapping/paging, or you run out of real memory. So > consider removing this option to give the system enough room or upgrade > to a machine with \> 4 GB RAM. Another option is adding more nodes of > course. > > If you stay with 2 GB RAM I would also recommend to disable 64-bit > server JVM with "java -d32 -client" in order to save some bits on the > heap. 64-bit server JVM makes sense on machines where you can use > virtual address space in the area of \> 4GB RAM. > > Jörg > > Am 09.04.13 12:07, schrieb Talha Khan: > > > I have set up a 1 node cluster of ES using ES version 0.19.11 for > > storing some logs. After inserting about 5000 logs I now see that my > > CPU usage is rocketing above 100% constanltly even when no/very few > > search queries are reaching it. ES has become very unresponsive and > > some of my search queries randomly return the following exception. > > "error" : "EsRejectedExecutionException[rejected execution of > > [org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction$2]]", > > > "status" : 503 > > > > By randomly i mean sometimes they succeed but mostly they fail. > > > > Here is my ES config: > > > > index.cache.field.expire: 20m > > index.cache.field.type: soft > > > > bootstrap.mlockall: true > > threadpool: > > search: > > type: fixed > > size: 20 > > queue\_size: 1000 > > index: > > type: fixed > > size: 16 > > bulk: > > type: fixed > > size: 5 > > > > I did some digging into the issue. The stats for node\_stats, > > hot\_threads api and jstack can be found in this gist > > > > [Elasticsearch high cpu · GitHub](https://gist.github.com/ahmedtalhakhan/5343784) > > > > I am also attaching the results of bigdesk. The cluster health is > > green but looking at these results, it seems that ES is very low on > > heap memory. But i am unable to understand the constant high CPU usage > > even in the idle state. How can this issue be solved? > > > > Kindly give some context and explanation to any answer. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:41am UTC](https://discuss.elastic.co/t/very-high-cpu-usage-of-es-in-idle-state/11499/4 "2017-07-06T02:41:57Z")