# Degrading performance, weird 100%CPU **URL:** \ **Category:** Elasticsearch\ **Created:** [August 31, 2013, 10:18pm UTC](https://discuss.elastic.co/t/degrading-performance-weird-100-cpu/13422 "2013-08-31T22:18:04Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![AlexeyV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexeyv/32/2126_2.png) [@AlexeyV](https://discuss.elastic.co/u/AlexeyV)\ **Post date:** [August 31, 2013, 10:18pm UTC](https://discuss.elastic.co/t/degrading-performance-weird-100-cpu/13422/1 "2013-08-31T22:18:04Z")
Hi there, i've got a 3 node cluster (5 shard +1 replica ) index, which is 1GB in size. I've got some GEO docs (2.5 million) in my index. Documents contain geo\_point data, which i query for using geo\_bbox filter with a specific bounding box. For a given test, i have a query that returns 0 rows and takes 40ms to complete and barely noticable CPU usage at all. When i crank up service calls (50 concurrent threads), i notice all my 3 nodes start hitting 100% cpu, and query degrades to 200ms , 500ms+ up to few minutes. This test is done via hitting of an application service which uses Java SDK to issue a query using TransportClient, which is configured to use all 3 nodes. My question is, why am i getting a degraded performance for the same query? I would assume it should if anything just pull it out of cache (note result returned by query is intentionally zero documents). I'm suspecting theres another factor that overloads my ES cluster, perhaps something outside of query that i am missing? Any advice is greatly appretiated. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 1, 2013, 5:26am UTC](https://discuss.elastic.co/t/degrading-performance-weird-100-cpu/13422/2 "2013-09-01T05:26:24Z")
Here's an extract of doc: [http://www.elasticsearch.org/guide/reference/query-dsl/geo-bounding-box-filter/](http://www.elasticsearch.org/guide/reference/query-dsl/geo-bounding-box-filter/) The result of the filter is not cached by default. The \_cache can be set to true to cache the result of the filter. This is handy when the same bounding box parameters are used on several (many) other queries. Note, the process of caching the first execution is higher when caching (since it needs to satisfy different queries). Does it help? -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 1 sept. 2013 à 00:18, Alexey Volochenko [alexey.volochenko@gmail.com](mailto:alexey.volochenko@gmail.com) a écrit : Hi there, i've got a 3 node cluster (5 shard +1 replica ) index, which is 1GB in size. I've got some GEO docs (2.5 million) in my index. Documents contain geo\_point data, which i query for using geo\_bbox filter with a specific bounding box. For a given test, i have a query that returns 0 rows and takes 40ms to complete and barely noticable CPU usage at all. When i crank up service calls (50 concurrent threads), i notice all my 3 nodes start hitting 100% cpu, and query degrades to 200ms , 500ms+ up to few minutes. This test is done via hitting of an application service which uses Java SDK to issue a query using TransportClient, which is configured to use all 3 nodes. My question is, why am i getting a degraded performance for the same query? I would assume it should if anything just pull it out of cache (note result returned by query is intentionally zero documents). I'm suspecting theres another factor that overloads my ES cluster, perhaps something outside of query that i am missing? ## Any advice is greatly appretiated. You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![AlexeyV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexeyv/32/2126_2.png) [@AlexeyV](https://discuss.elastic.co/u/AlexeyV)\ **Post date:** [September 3, 2013, 10:10pm UTC](https://discuss.elastic.co/t/degrading-performance-weird-100-cpu/13422/3 "2013-09-03T22:10:46Z")
Thanks David! i will try this out and post results. On Saturday, August 31, 2013 10:26:24 PM UTC-7, David Pilato wrote: > Here's an extract of doc: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/geo-bounding-box-filter/) > > The result of the filter is not cached by default. The \_cache can be set > to true to cache the _result_ of the filter. This is handy when the same > bounding box parameters are used on several (many) other queries. Note, the > process of caching the first execution is higher when caching (since it > needs to satisfy different queries). > > Does it help? > > -- > David 😉 > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > Le 1 sept. 2013 à 00:18, Alexey Volochenko \<[alexey.v...@gmail.com](mailto:alexey.v...@gmail.com)\\> > a écrit : > > Hi there, i've got a 3 node cluster (5 shard +1 replica ) index, which is > 1GB in size. > I've got some GEO docs (2.5 million) in my index. Documents contain > geo\_point data, which i query for using geo\_bbox filter with a specific > bounding box. For a given test, i have a query that returns 0 rows and > takes 40ms to complete and barely noticable CPU usage at all. When i crank > up service calls (50 concurrent threads), i notice all my 3 nodes start > hitting 100% cpu, and query degrades to 200ms , 500ms+ up to few minutes. > This test is done via hitting of an application service which uses Java SDK > to issue a query using TransportClient, which is configured to use all 3 > nodes. > > My question is, why am i getting a degraded performance for the same > query? I would assume it should if anything just pull it out of cache (note > result returned by query is intentionally zero documents). I'm suspecting > theres another factor that overloads my ES cluster, perhaps something > outside of query that i am missing? > > Any advice is greatly appretiated. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![AlexeyV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexeyv/32/2126_2.png) [@AlexeyV](https://discuss.elastic.co/u/AlexeyV)\ **Post date:** [September 4, 2013, 11:31pm UTC](https://discuss.elastic.co/t/degrading-performance-weird-100-cpu/13422/4 "2013-09-04T23:31:18Z")
So i've done more testing, and seems like caching didnt help, though even if it did i kind of doubt it would be a solution due to nature of the bounding box randomness + extra hit on first query. I ended up spinning up beefier server with more CPU capacity to address the performance degregation. Will continue looking for ways to reduce cpu use. On Tuesday, September 3, 2013 3:10:46 PM UTC-7, Alexey Volochenko wrote: > Thanks David! i will try this out and post results. > > On Saturday, August 31, 2013 10:26:24 PM UTC-7, David Pilato wrote: > > > Here's an extract of doc: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/query-dsl/geo-bounding-box-filter/) > > > > The result of the filter is not cached by default. The \_cache can be set > > to true to cache the _result_ of the filter. This is handy when the same > > bounding box parameters are used on several (many) other queries. Note, the > > process of caching the first execution is higher when caching (since it > > needs to satisfy different queries). > > > > Does it help? > > > > -- > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs > > > > Le 1 sept. 2013 à 00:18, Alexey Volochenko [alexey.v...@gmail.com](mailto:alexey.v...@gmail.com) a > > écrit : > > > > Hi there, i've got a 3 node cluster (5 shard +1 replica ) index, which is > > 1GB in size. > > I've got some GEO docs (2.5 million) in my index. Documents contain > > geo\_point data, which i query for using geo\_bbox filter with a specific > > bounding box. For a given test, i have a query that returns 0 rows and > > takes 40ms to complete and barely noticable CPU usage at all. When i crank > > up service calls (50 concurrent threads), i notice all my 3 nodes start > > hitting 100% cpu, and query degrades to 200ms , 500ms+ up to few minutes. > > This test is done via hitting of an application service which uses Java SDK > > to issue a query using TransportClient, which is configured to use all 3 > > nodes. > > > > My question is, why am i getting a degraded performance for the same > > query? I would assume it should if anything just pull it out of cache (note > > result returned by query is intentionally zero documents). I'm suspecting > > theres another factor that overloads my ES cluster, perhaps something > > outside of query that i am missing? > > > > Any advice is greatly appretiated. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:18am UTC](https://discuss.elastic.co/t/degrading-performance-weird-100-cpu/13422/5 "2017-07-06T02:18:07Z")