Load doesnt spread even on all nodes in cluster

Elastic Stack Elasticsearch
1

Hi,

I have a cluster of three nodes. indesx setup with 12 shards and 2 replica.
While running a load test for search only queries, I notce that the load doesn't spread the same on all nodes. I see two nodes with load ~6 and onde node get load as high as 15 to 20.

Notes:

  1. Same results with / without route
  2. HTTP Queries are load balanced with random node selection
  3. Using an addiotional client node doesn't change anything
  4. No indexing / inserts are done
  5. The loaded node changes on every new load test that we start
  6. The data on the nodes are spread eventliy

Any idea why this might happn?
Thanks

2

Those are all things I'd ask first. If you've made sure those are all right I'd:

  1. Use the hot_threads api and compare.
  2. Compare the output of sudo -u elasticsearch bash -c 'jstack `cat /var/run/elasticsearch.pid`'
    the loaded and unloaded nodes. Repeat this command a few times so you can get a sense of what is going on.
  3. Make double sure the data really is distributed evenly and you aren't using routing.

Note that having 12 shards and 2 replicas with three nodes isn't going to be as efficient on query as having 1 shard and 2 replicas.