Increase CPU utilization without increasing shard count

Elastic Stack Elasticsearch
1

Updated with more details (11/3/2015)

Hi, according to this post by Eric Gaumer, dated two years ago, to increase CPU utilization one needs to increase the number of shards. I wonder if it is now possible to use all 32 cores on an AWS c3.8xlarge for indexing while keeping only one shard per node.

  • With one shard, I can get to about 33% CPU usage (~12 cores).
  • With three shards, I can get to about 60% CPU usage (~18 cores).

Thanks!

2

Lucene is multithreaded, but that's about all I know in that aspect sorry.
Maybe someone with more knowledge can comment :slight_smile:

3

Since you are asking this question you must have some kind of problem.

In my experience, Elasticsearch is using all cores available.

The problem (at least for me) is not that the cores are not used - the bottleneck is my storage.

The load if my server can be ~20, but the cpu usage is at 5%. (20 core server)

Hope this helps

4

@lwintergerst Please see my updated question. With one shard, I thought indexing was IO-bound. But when I increased shard count to three, there's a significant increase in CPU usage (almost doubled: 33 -> 60). Other indicators (index rate, IOPS, cluster load) are also up by a lot.