Higher CPU usage in cluster than a single node

I am operating an Elasticsearch single node and Kibana in a prod to monitor access log and WAS log. But some queries of Kibana dashboard were too late ( about over 7 sec ) to get information into the dashboard. So we are considering to set up 3-nodes Elastic cluster and have installed 3-nodes ES ( 1 master ) in staging servers.

As a result of the test of executing a slow query in staging servers( 3-nodes and single node), although the response time was shorten to 3-4 sec, CPU usage was far higher than a single node. Comparison is table below.

response time CPU usage
single node 6 - 7 sec under 30%
3-node 3 - 4 sec all nodes over 50%, even 70-80%

All servers have 4-core, 16GB and I have set -Xms 2G, -Xmx 2G in a jvm option configure file.
Both the single node and the cluster have the same indices and 5 primary shards and 1 replica.( So, the indices in the single node have yellow status.. ) The size of each index is about 1 ~ 2 GB.

Since the query has many 'OR' condition, I admit that it is needed to optimize that. But It seems to be weired that all nodes in cluster have high CPU usage about 3 times compared to a single node nevertheless they have the same configuration and spec.

  1. Is it normal that CPU usage is high in multiple nodes cluster?
  2. Or do I have things to optimize the cluster or change configurations?

Thank you for replies in advance!!

1 Like