# Is 3k search/sec high volumn? (High CPU usage)

**URL:** https://discuss.elastic.co/t/is-3k-search-sec-high-volumn-high-cpu-usage/35341
**Category:** Elasticsearch
**Created:** [November 23, 2015, 6:20pm UTC](https://discuss.elastic.co/t/is-3k-search-sec-high-volumn-high-cpu-usage/35341 "2015-11-23T18:20:22Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![essis](https://avatars.discourse-cdn.com/v4/letter/e/919ad9/32.png) [@essis](https://discuss.elastic.co/u/essis)
#### Post date: [November 23, 2015, 6:20pm UTC](https://discuss.elastic.co/t/is-3k-search-sec-high-volumn-high-cpu-usage/35341/1 "2015-11-23T18:20:22Z")

</div>

Hi  
I have Elasticsearch 8 nodes cluster of 6 to 8 CPUs with 16GB memory and SSDs in them.  
I am doing search query on the index which has 2 million documents in 8 shard + 1 replica. Document size is moderate so index size is around 2GB.  
Current traffic makes 3000 search query per second, and overall CPU usage is around 50%. And, if the query rate goes up over 4000/s then some nodes reach 100% and start dropping queues which causes application failure.  
There's no indexing during the period.  
Each query takes less than 50ms. I tried to optimize search query, but simple match all query also takes almost half of current usage, which is still too high.  
One interesting thing is that if I optimize index with `max_num_segments=1` then CPU usage goes down to a half. So I reduced `segments_per_tier` to 3 but it didn't help.  
Is this normal capacity of elasticsearch? Or is there something wrong with my cluster.  
I used both Oracle and OpenJDK, and result is similar on both.  
This is hot thread dump.

> <https://gist.github.com/janghwan/351b1ed8b0f315361890>

---

<div class="post-metadata">

### Author: ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)
#### Post date: [November 23, 2015, 10:27pm UTC](https://discuss.elastic.co/t/is-3k-search-sec-high-volumn-high-cpu-usage/35341/2 "2015-11-23T22:27:51Z")

</div>

> [@essis](#):
>
> So I reduced segments\_per\_tier to 3 but it didn't help.

Depending on the version this doesn't kick in properly after the index is created. I don't have a link.

> [@essis](#):
>
> Is this normal capacity of elasticsearch? Or is there something wrong with my cluster.

What you describe is fairly normal for when the cluster is at the edge.

Your index is fairly small so I'm not surprised I don't see IO load.

The hot\_threads isn't doing well. It doesn't do a good job when you have many short running jobs. Your best bet is to use `jstack` on a node several times in a row while its under load and analyze that.

You'll have to post example search queries for us to help with those. Depending on what you are doing match all might not be a great indicator. Like if fetching from \_source is taking a while then match\_all isn't going to change anything. Really the stack traces are you best bet for figuring out what is up.

Another thing to check is `jstat gcutil <pid> 3s 100`. You can use that to figure out how much time is being taken up by gc. Its harder to figure out what is taking up the memory, but with the queries you could probably puzzle it out.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 11:36pm UTC](https://discuss.elastic.co/t/is-3k-search-sec-high-volumn-high-cpu-usage/35341/3 "2017-07-05T23:36:25Z")

</div>


