# High CPU load during search(elasticsearch 1.2.1) **URL:** \ **Category:** Elasticsearch\ **Created:** [September 4, 2014, 2:51pm UTC](https://discuss.elastic.co/t/high-cpu-load-during-search-elasticsearch-1-2-1/19624 "2014-09-04T14:51:08Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Anton\_A](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@Anton\_A](https://discuss.elastic.co/u/Anton_A)\ **Post date:** [September 4, 2014, 2:51pm UTC](https://discuss.elastic.co/t/high-cpu-load-during-search-elasticsearch-1-2-1/19624/1 "2014-09-04T14:51:08Z")
Hi, Everyone. I have no huge experience with elsticsearch but meet performance problem on our environment and need somehow to resolve it or figure out what is the problem. Some background: we have multiple environments with the same configuration and only one have this issue. We using elastic search 1.2.1. We doing daily job and it fire each one or two seconds or more often search requests to es. First time we doing it, it finished successfully in 3h next day we fire it and it finished in 6 hours and next time it not finished in 9 hours after that we performing restart and everything became normal, but than it happens again in again. Symptoms: 1. We have 16 core CPU and all cores are 90% loaded. after job finished usage dropped to 4%. 2. Memory consumption in JVM not more than 50% at that moment 3. We have 70 search threads and only 20 of them working at that moment 4. Attached result of 2 hotthreads request from different days. 5. Here is JVM value from node stats: jvm: { Realy appreciate any advices how to handle this issue and ready provide any data. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 4, 2014, 5:51pm UTC](https://discuss.elastic.co/t/high-cpu-load-during-search-elasticsearch-1-2-1/19624/2 "2014-09-04T17:51:23Z")
In your hot threads dump, you see the culprit, it has something to do with a plugin you use, not with Elasticsearch. com.clarabridge.elasticsearch.facet.sampling Ask the people who provided you with this software. Jörg On Thu, Sep 4, 2014 at 4:51 PM, Anton A [zer0orama@gmail.com](mailto:zer0orama@gmail.com) wrote: > Hi, Everyone. I have no huge experience with elsticsearch but meet > performance problem on our environment and need somehow to resolve it or > figure out what is the problem. Some background: we have multiple > environments with the same configuration and only one have this issue. We > using Elasticsearch 1.2.1. We doing daily job and it fire each one or two > seconds or more often search requests to es. First time we doing it, it > finished successfully in 3h next day we fire it and it finished in 6 hours > and next time it not finished in 9 hours after that we performing restart > and everything became normal, but than it happens again in again. > Symptoms: > > 1. We have 16 core CPU and all cores are 90% loaded. after job finished > usage dropped to 4%. > > 2. Memory consumption in JVM not more than 50% at that moment > > 3. We have 70 search threads and only 20 of them working at that moment > > 4. Attached result of 2 hotthreads request from different days. > > 5. Here is JVM value from node stats: > jvm: { > > Realy appreciate any advices how to handle this issue and ready provide > any data. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH6%2BM3xcD6tcOXudeWPRYj3jhGj6wDUg-gZ-%2B%2BAF9i8fg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoH6%2BM3xcD6tcOXudeWPRYj3jhGj6wDUg-gZ-%2B%2BAF9i8fg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Anton\_A](https://avatars.discourse-cdn.com/v4/letter/a/87869e/32.png) [@Anton\_A](https://discuss.elastic.co/u/Anton_A)\ **Post date:** [September 5, 2014, 8:34am UTC](https://discuss.elastic.co/t/high-cpu-load-during-search-elasticsearch-1-2-1/19624/3 "2014-09-05T08:34:40Z")
Hi, Jörg. Thanks for replay, as I said before we using this on multiply instances and met this problem only one particular one. I removed this plugin and checked again, this won't help. Here is hotthreads from this run. I realy appretiate if you suggest next steps what we can look. Thanks. четверг, 4 сентября 2014 г., 20:51:30 UTC+3 пользователь Jörg Prante написал: > In your hot threads dump, you see the culprit, it has something to do with > a plugin you use, not with Elasticsearch. > > com.clarabridge.elasticsearch.facet.sampling > > Ask the people who provided you with this software. > > Jörg > > On Thu, Sep 4, 2014 at 4:51 PM, Anton A \<[zer0...@gmail.com](mailto:zer0...@gmail.com) \\> > wrote: > > > Hi, Everyone. I have no huge experience with elsticsearch but meet > > performance problem on our environment and need somehow to resolve it or > > figure out what is the problem. Some background: we have multiple > > environments with the same configuration and only one have this issue. We > > using Elasticsearch 1.2.1. We doing daily job and it fire each one or two > > seconds or more often search requests to es. First time we doing it, it > > finished successfully in 3h next day we fire it and it finished in 6 hours > > and next time it not finished in 9 hours after that we performing restart > > and everything became normal, but than it happens again in again. > > Symptoms: > > > > 1. We have 16 core CPU and all cores are 90% loaded. after job finished > > usage dropped to 4%. > > > > 2. Memory consumption in JVM not more than 50% at that moment > > > > 3. We have 70 search threads and only 20 of them working at that moment > > > > 4. Attached result of 2 hotthreads request from different days. > > > > 5. Here is JVM value from node stats: > > jvm: { > > > > Realy appreciate any advices how to handle this issue and ready provide > > any data. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e0c315cb-6f90-4270-88a5-61788502a0a6%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e0c315cb-6f90-4270-88a5-61788502a0a6%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![PShah](https://avatars.discourse-cdn.com/v4/letter/p/a698b9/32.png) [@PShah](https://discuss.elastic.co/u/PShah)\ **Post date:** [September 5, 2014, 1:04pm UTC](https://discuss.elastic.co/t/high-cpu-load-during-search-elasticsearch-1-2-1/19624/4 "2014-09-05T13:04:00Z")
Hi Jorg, Anton is right we removed the plugin and double checked ES is taking up our bulk of the time. We do see that number of evictions are high filter\_cache: { memory\_size\_in\_bytes: 10508060 evictions: 0 } id\_cache: { memory\_size\_in\_bytes: 276840500 } fielddata: { memory\_size\_in\_bytes: 416181852 evictions: 9842 } the fielddata cache is set to 40%. Do you think the number of evictions we have are the cause of low performance ? If yes, how can we reduce it ? On Friday, September 5, 2014 4:34:41 AM UTC-4, Anton A wrote: > Hi, Jörg. Thanks for replay, as I said before we using this on multiply > instances and met this problem only one particular one. I removed this > plugin and checked again, this won't help. Here is hotthreads from this > run. I realy appretiate if you suggest next steps what we can look. > Thanks. > > четверг, 4 сентября 2014 г., 20:51:30 UTC+3 пользователь Jörg Prante > написал: > > > In your hot threads dump, you see the culprit, it has something to do > > with a plugin you use, not with Elasticsearch. > > > > com.clarabridge.elasticsearch.facet.sampling > > > > Ask the people who provided you with this software. > > > > Jörg > > > > On Thu, Sep 4, 2014 at 4:51 PM, Anton A [zer0...@gmail.com](mailto:zer0...@gmail.com) wrote: > > > > > Hi, Everyone. I have no huge experience with elsticsearch but meet > > > performance problem on our environment and need somehow to resolve it or > > > figure out what is the problem. Some background: we have multiple > > > environments with the same configuration and only one have this issue. We > > > using Elasticsearch 1.2.1. We doing daily job and it fire each one or two > > > seconds or more often search requests to es. First time we doing it, it > > > finished successfully in 3h next day we fire it and it finished in 6 hours > > > and next time it not finished in 9 hours after that we performing restart > > > and everything became normal, but than it happens again in again. > > > Symptoms: > > > > > > 1. We have 16 core CPU and all cores are 90% loaded. after job finished > > > usage dropped to 4%. > > > > > > 2. Memory consumption in JVM not more than 50% at that moment > > > > > > 3. We have 70 search threads and only 20 of them working at that moment > > > > > > 4. Attached result of 2 hotthreads request from different days. > > > > > > 5. Here is JVM value from node stats: > > > jvm: { > > > > > > Realy appreciate any advices how to handle this issue and ready provide > > > any data. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [September 7, 2014, 1:36pm UTC](https://discuss.elastic.co/t/high-cpu-load-during-search-elasticsearch-1-2-1/19624/5 "2014-09-07T13:36:17Z")
The filter cache, which is very fast, is almost not used, only 10m. The field data cache is highly used, with 416m. But 416m is no problem for ES. The "hot threads" show that almost all threads are busy with calculating scores. This is typical for complex queries. Computing relevancy for scores means that all docs have to be visited in the result set, but this can often be optimized by rewriting queries. I assume you use just queries and very few filters. Maybe you can rewrite queries to use filters? This will give a huge performance boost. Jörg On Fri, Sep 5, 2014 at 3:04 PM, PShah [perks5@gmail.com](mailto:perks5@gmail.com) wrote: > Hi Jorg, > > Anton is right we removed the plugin and double checked ES is taking up > our bulk of the time. We do see that number of evictions are high > > filter\_cache: { > memory\_size\_in\_bytes: 10508060 > evictions: 0 > } > id\_cache: { > memory\_size\_in\_bytes: 276840500 > } > fielddata: { > memory\_size\_in\_bytes: 416181852 > evictions: 9842 > } > > the fielddata cache is set to 40%. Do you think the number of evictions we > have are the cause of low performance ? If yes, how can we reduce it ? > > On Friday, September 5, 2014 4:34:41 AM UTC-4, Anton A wrote: > > > Hi, Jörg. Thanks for replay, as I said before we using this on multiply > > instances and met this problem only one particular one. I removed this > > plugin and checked again, this won't help. Here is hotthreads from this > > run. I realy appretiate if you suggest next steps what we can look. > > Thanks. > > > > четверг, 4 сентября 2014 г., 20:51:30 UTC+3 пользователь Jörg Prante > > написал: > > > > > In your hot threads dump, you see the culprit, it has something to do > > > with a plugin you use, not with Elasticsearch. > > > > > > com.clarabridge.elasticsearch.facet.sampling > > > > > > Ask the people who provided you with this software. > > > > > > Jörg > > > > > > On Thu, Sep 4, 2014 at 4:51 PM, Anton A [zer0...@gmail.com](mailto:zer0...@gmail.com) wrote: > > > > > > > Hi, Everyone. I have no huge experience with elsticsearch but meet > > > > performance problem on our environment and need somehow to resolve it or > > > > figure out what is the problem. Some background: we have multiple > > > > environments with the same configuration and only one have this issue. We > > > > using Elasticsearch 1.2.1. We doing daily job and it fire each one or two > > > > seconds or more often search requests to es. First time we doing it, it > > > > finished successfully in 3h next day we fire it and it finished in 6 hours > > > > and next time it not finished in 9 hours after that we performing restart > > > > and everything became normal, but than it happens again in again. > > > > Symptoms: > > > > > > > > 1. We have 16 core CPU and all cores are 90% loaded. after job finished > > > > usage dropped to 4%. > > > > > > > > 2. Memory consumption in JVM not more than 50% at that moment > > > > > > > > 3. We have 70 search threads and only 20 of them working at that moment > > > > > > > > 4. Attached result of 2 hotthreads request from different days. > > > > > > > > 5. Here is JVM value from node stats: > > > > jvm: { > > > > > > > > Realy appreciate any advices how to handle this issue and ready provide > > > > any data. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18% > > > > [40googlegroups.com](http://40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEqF8-T7%2B1cwpknPWDhmW8ZAhYetJEfq%3D%3Dy02eFVW%2Bhrg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEqF8-T7%2B1cwpknPWDhmW8ZAhYetJEfq%3D%3Dy02eFVW%2Bhrg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![PShah](https://avatars.discourse-cdn.com/v4/letter/p/a698b9/32.png) [@PShah](https://discuss.elastic.co/u/PShah)\ **Post date:** [October 23, 2014, 5:58pm UTC](https://discuss.elastic.co/t/high-cpu-load-during-search-elasticsearch-1-2-1/19624/6 "2014-10-23T17:58:53Z")
Hi Jörg, The problem we are trying to describe here is that why the query performance degrades over time. If I restart my service, the performance comes back to normal but after some days the same queries run slower. I am not sure if changing the queries would help in that case. We even experimented with simple queries and it all slows down. On Sunday, September 7, 2014 9:36:30 AM UTC-4, Jörg Prante wrote: > The filter cache, which is very fast, is almost not used, only 10m. > > The field data cache is highly used, with 416m. But 416m is no problem for > ES. > > The "hot threads" show that almost all threads are busy with calculating > scores. This is typical for complex queries. Computing relevancy for scores > means that all docs have to be visited in the result set, but this can > often be optimized by rewriting queries. > > I assume you use just queries and very few filters. Maybe you can rewrite > queries to use filters? This will give a huge performance boost. > > Jörg > > On Fri, Sep 5, 2014 at 3:04 PM, PShah \<[per...@gmail.com](mailto:per...@gmail.com) \\> > wrote: > > > Hi Jorg, > > > > Anton is right we removed the plugin and double checked ES is taking up > > our bulk of the time. We do see that number of evictions are high > > > > filter\_cache: { > > memory\_size\_in\_bytes: 10508060 > > evictions: 0 > > } > > id\_cache: { > > memory\_size\_in\_bytes: 276840500 > > } > > fielddata: { > > memory\_size\_in\_bytes: 416181852 > > evictions: 9842 > > } > > > > the fielddata cache is set to 40%. Do you think the number of evictions > > we have are the cause of low performance ? If yes, how can we reduce it ? > > > > On Friday, September 5, 2014 4:34:41 AM UTC-4, Anton A wrote: > > > > > Hi, Jörg. Thanks for replay, as I said before we using this on multiply > > > instances and met this problem only one particular one. I removed this > > > plugin and checked again, this won't help. Here is hotthreads from this > > > run. I realy appretiate if you suggest next steps what we can look. > > > Thanks. > > > > > > четверг, 4 сентября 2014 г., 20:51:30 UTC+3 пользователь Jörg Prante > > > написал: > > > > > > > In your hot threads dump, you see the culprit, it has something to do > > > > with a plugin you use, not with Elasticsearch. > > > > > > > > com.clarabridge.elasticsearch.facet.sampling > > > > > > > > Ask the people who provided you with this software. > > > > > > > > Jörg > > > > > > > > On Thu, Sep 4, 2014 at 4:51 PM, Anton A [zer0...@gmail.com](mailto:zer0...@gmail.com) wrote: > > > > > > > > > Hi, Everyone. I have no huge experience with elsticsearch but meet > > > > > performance problem on our environment and need somehow to resolve it or > > > > > figure out what is the problem. Some background: we have multiple > > > > > environments with the same configuration and only one have this issue. We > > > > > using Elasticsearch 1.2.1. We doing daily job and it fire each one or two > > > > > seconds or more often search requests to es. First time we doing it, it > > > > > finished successfully in 3h next day we fire it and it finished in 6 hours > > > > > and next time it not finished in 9 hours after that we performing restart > > > > > and everything became normal, but than it happens again in again. > > > > > Symptoms: > > > > > > > > > > 1. We have 16 core CPU and all cores are 90% loaded. after job > > > > > finished usage dropped to 4%. > > > > > > > > > > 2. Memory consumption in JVM not more than 50% at that moment > > > > > > > > > > 3. We have 70 search threads and only 20 of them working at that moment > > > > > > > > > > 4. Attached result of 2 hotthreads request from different days. > > > > > > > > > > 5. Here is JVM value from node stats: > > > > > jvm: { > > > > > > > > > > Realy appreciate any advices how to handle this issue and ready > > > > > provide any data. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18% > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/33ce9d29-69c0-4f78-aa27-ea625da590e0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/33ce9d29-69c0-4f78-aa27-ea625da590e0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![PShah](https://avatars.discourse-cdn.com/v4/letter/p/a698b9/32.png) [@PShah](https://discuss.elastic.co/u/PShah)\ **Post date:** [October 23, 2014, 6:04pm UTC](https://discuss.elastic.co/t/high-cpu-load-during-search-elasticsearch-1-2-1/19624/7 "2014-10-23T18:04:39Z")
This is indicative of some sort of resource leak. I checked heap, CPU, nothing seems to stand out. > Hi Jörg, > > The problem we are trying to describe here is that why the query > performance degrades over time. If I restart my service, the performance > comes back to normal but after some days the same queries run slower. I am > not sure if changing the queries would help in that case. We even > experimented with simple queries and it all slows down. > > On Sunday, September 7, 2014 9:36:30 AM UTC-4, Jörg Prante wrote: > > > The filter cache, which is very fast, is almost not used, only 10m. > > > > The field data cache is highly used, with 416m. But 416m is no problem > > for ES. > > > > The "hot threads" show that almost all threads are busy with calculating > > scores. This is typical for complex queries. Computing relevancy for scores > > means that all docs have to be visited in the result set, but this can > > often be optimized by rewriting queries. > > > > I assume you use just queries and very few filters. Maybe you can rewrite > > queries to use filters? This will give a huge performance boost. > > > > Jörg > > > > On Fri, Sep 5, 2014 at 3:04 PM, PShah [per...@gmail.com](mailto:per...@gmail.com) wrote: > > > > > Hi Jorg, > > > > > > Anton is right we removed the plugin and double checked ES is taking up > > > our bulk of the time. We do see that number of evictions are high > > > > > > filter\_cache: { > > > memory\_size\_in\_bytes: 10508060 > > > evictions: 0 > > > } > > > id\_cache: { > > > memory\_size\_in\_bytes: 276840500 > > > } > > > fielddata: { > > > memory\_size\_in\_bytes: 416181852 > > > evictions: 9842 > > > } > > > > > > the fielddata cache is set to 40%. Do you think the number of evictions > > > we have are the cause of low performance ? If yes, how can we reduce it ? > > > > > > On Friday, September 5, 2014 4:34:41 AM UTC-4, Anton A wrote: > > > > > > > Hi, Jörg. Thanks for replay, as I said before we using this on multiply > > > > instances and met this problem only one particular one. I removed this > > > > plugin and checked again, this won't help. Here is hotthreads from this > > > > run. I realy appretiate if you suggest next steps what we can look. > > > > Thanks. > > > > > > > > четверг, 4 сентября 2014 г., 20:51:30 UTC+3 пользователь Jörg Prante > > > > написал: > > > > > > > > > In your hot threads dump, you see the culprit, it has something to do > > > > > with a plugin you use, not with Elasticsearch. > > > > > > > > > > com.clarabridge.elasticsearch.facet.sampling > > > > > > > > > > Ask the people who provided you with this software. > > > > > > > > > > Jörg > > > > > > > > > > On Thu, Sep 4, 2014 at 4:51 PM, Anton A [zer0...@gmail.com](mailto:zer0...@gmail.com) wrote: > > > > > > > > > > > Hi, Everyone. I have no huge experience with elsticsearch but meet > > > > > > performance problem on our environment and need somehow to resolve it or > > > > > > figure out what is the problem. Some background: we have multiple > > > > > > environments with the same configuration and only one have this issue. We > > > > > > using Elasticsearch 1.2.1. We doing daily job and it fire each one or two > > > > > > seconds or more often search requests to es. First time we doing it, it > > > > > > finished successfully in 3h next day we fire it and it finished in 6 hours > > > > > > and next time it not finished in 9 hours after that we performing restart > > > > > > and everything became normal, but than it happens again in again. > > > > > > Symptoms: > > > > > > > > > > > > 1. We have 16 core CPU and all cores are 90% loaded. after job > > > > > > finished usage dropped to 4%. > > > > > > > > > > > > 2. Memory consumption in JVM not more than 50% at that moment > > > > > > > > > > > > 3. We have 70 search threads and only 20 of them working at that > > > > > > moment > > > > > > > > > > > > 4. Attached result of 2 hotthreads request from different days. > > > > > > > > > > > > 5. Here is JVM value from node stats: > > > > > > jvm: { > > > > > > > > > > > > Realy appreciate any advices how to handle this issue and ready > > > > > > provide any data. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > > > msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18% > > > > > > [40googlegroups.com](http://40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ee66e9cc-1f7c-498b-b67e-e896f5656c18%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/6f1f043f-9b50-475f-b0d0-45fec51abba0%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3add7bca-ea4b-4f4c-a3ab-abd27967949a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3add7bca-ea4b-4f4c-a3ab-abd27967949a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:54am UTC](https://discuss.elastic.co/t/high-cpu-load-during-search-elasticsearch-1-2-1/19624/8 "2017-07-06T00:54:08Z")