I am operating an Elasticsearch single node and Kibana in a prod to monitor access log and WAS log. But some queries of Kibana dashboard were too late ( about over 7 sec ) to get information into the dashboard. So we are considering to set up 3-nodes Elastic cluster and have installed 3-nodes ES ( 1 master ) in staging servers.
As a result of the test of executing a slow query in staging servers( 3-nodes and single node), although the response time was shorten to 3-4 sec, CPU usage was far higher than a single node. Comparison is table below.
| response time | CPU usage | |
|---|---|---|
| single node | 6 - 7 sec | under 30% |
| 3-node | 3 - 4 sec | all nodes over 50%, even 70-80% |
All servers have 4-core, 16GB and I have set -Xms 2G, -Xmx 2G in a jvm option configure file.
Both the single node and the cluster have the same indices and 5 primary shards and 1 replica.( So, the indices in the single node have yellow status.. ) The size of each index is about 1 ~ 2 GB.
Since the query has many 'OR' condition, I admit that it is needed to optimize that. But It seems to be weired that all nodes in cluster have high CPU usage about 3 times compared to a single node nevertheless they have the same configuration and spec.
- Is it normal that CPU usage is high in multiple nodes cluster?
- Or do I have things to optimize the cluster or change configurations?
Thank you for replies in advance!!