Hi all, I met a query performance issue when performance testing against Elastic Search V2.4.3:
1. When query performance testing against the cluster with 64 shards of one index( 64 = 16*4, 16 shards, 1 primary and 3 replication of each ) on 64 data nodes ( so one shard allocated on one data node ), everything works well as expected, we get QPS capacity as about 1800/Sec.
2. Keep everything the same as #1, I stopped ES service on one of the data node, and saw the shard on it auto-re-allocated to another data node ( the re-allocation took about 20 minutes in our test), so now:
a. we have 64 shards allocated on 63 data nodes
b. 62 data nodes have one single shard on each, 1 data node has two shards on it
Under circumstance of #2, I re-tested against the cluster with the same test settings including the same query data as #1, we get QPS capacity at about only 1000/Sec.
From the Marvel dashboard, I saw the CPU rate spikes on that data node with the two shards at over 95%, and the other nodes running just as well as #1, so I assume this data node with multiple shards can be the bottleneck of test #2.
The situation is:
we deployed ES cluster on Azure, and Azure will do maintaining work like OS upgrade now and then, so that some machines will just stop work, and the auto-re-allocation will take about 20 minutes, so there will be cases like test #2, and the big capacity drop is unacceptable to the customer traffic.
Questions:
1. Are we missing anything like ES configuration causing this performance issue? How can we avoid this kind of performance issue? Any advice?
2. We cannot understand: why only one more shard on one single data node causes this big capacity drop? We have 8 cores on each data node.
When a node fails, the node the shard is relocated to basically gets twice the load, which I suppose will slow down a significant portion of the queries. If you had more shards per node, the addition of another shard due to failure would have a smaller impact as the data volume would increase by less than 100%. I also wonder what impact adding another (empty) data node to the cluster would have. Would this get the shard that need to be relocated, leaving all nodes with exactly one shards? If this is what happens (I am not sure how Elasticsearch would handle this) it would also have the benefit of avoiding another relocation once the missing node comes back. The empty node would basically act as a client node ready to take on data if needed.
How much RAM do you have on each node? What is your average shard size?
RAM on each node is 56 GB and average shard size is about 30GB, and talk about memory, we sometimes saw the memory usage by the elastic search process is about 55GB on multiple shards node, only service restarting can get it down, which was another issue in the test.
Adding empty data node was one of our options, but the shard auto-re-allocation will take about 20 minutes, which is too long for the cluster to deal with the traffic during it.
Have you identified what is currently limiting performance? Is it CPU load? Network performance? I assume it is not disk I/O as you only have a 3GB shard per node, which should be cached by the OS. What does your heap usage look like? What kind of instance type is your cluster deployed on?
Sorry Christian, I made a typo here.. it is 30GB shard, the limitation is CPU, the multiple shards node CPU usage was over 95%, and the memory usage in this test is stable at about 30%
For this index, documents count is 664.7 million with total 64 shards.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.