Unbalanced primary shards affects search queries?

[ragian]

In a lot of topic is said that we shouldn't worry about balancing primary shards across all nodes.

Currently my primaries are highly unbalanced in my elasticsearch cluster (3 nodes with elastic 6.4.0).

On the cluster are performed mainly read/search queries and according to the documentation they
should balanced between the node of the shard, but it seems that they are performed only
on the server 3 (that one with high load).

curl -XGET 's01.node.consul:9200/_cat/thread_pool' | grep "search"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1845  100  1845    0     0  75275      0 --:--:-- --:--:-- --:--:-- 76875
"s01" search              0 0   2
"s02" search              0 0   0
"s03" search              7 8 164

This is one of the example of the queries:

GET host.elastic:9200/index_*/_search/?scroll=120s 
{
	"query": {
		"bool": {
			"must": [{
				"match": {
					"param1": "1111"
				}
			}, {
				"match": {
					"param2": "XXXXXX"
				}
			}]
		}
	},
	"sort": [{
		"entered_date": {
			"order": "asc"
		}
	}],
	"size": 1000
}

[DavidTurner]

Searches behave no differently between primaries and replicas, so there's something else going on here.

The simplest way to find out which nodes are performing a search is probably to use the profile API. In the response each shard involved in the search is identified by a string of the form [2aE02wS1R8q_QFnYu6vDVQ][twitter][0] where the first part is the node ID. I expect if you run your search with the profiler then you will see it using a mix of nodes.