Does Elasticsearch Data Node full of replicas routing request to other nodes?

Elastic Stack Elasticsearch
1

If the cluster has 4 nodes and the index has 4 primary shards.
Each shard has 3 replicas, it means each node contains all shard data.

When search query is sent to a node,. does it route query to other nodes? or it depends?
Comparing to each node has only one single shard without replica.. which one is faster in term of search speed and indexing speed?

2

See the preference parameter of a search request, which also explains the default behaviour:

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-your-data.html#search-preference

3

You can test with the query [from these docs](You can test with this query, which if you send to your first node you can see which shards and nodes it'll use: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-shards.html), which if you send to your first node you can see which shards and nodes it'll use.

Generally unless you target a document ID/shard or other ways to route to a single shard, the coordinating node (the one your client talks to) must do a scatter/gather operation and send the query to all of an index's shards

The question is how it chooses, in your case among the 4 copies of each shard - this is called Adaptive Replica Selection with this nice doc on that.

My reading of that is that among other things it uses past performance as an input, so in theory of all queries were local and fast, it might pick the local shards and not bother with other shards.

But the queue matters, also so if the local node is busy, it may send to other nodes.

5

So the cluster set up that each node contains all the shards seems to have better search speed than each node contain single shard.. may we say that? (Considering the cluster has only one index)

6

Oh I found something interesting.. with ?_explain=true. you will easily see that result will come from different nodes.... hmm this is interesting.

7

Yeah, it's pretty dynamic and expects to run under heavy load and shifting queues, etc. so it likely moves around like any distributed dynamic system. I imagine it can oscillate also since this node may be loaded and slow so it uses other nodes, then this node gets fast so uses this node, etc. but on balance, it all balances out.

I'm writing a blog on search data flow and will dig into the code to see how it does this.

8

It is very interesting to know... for example
if we have 16 data nodes containing one single index
A. 8 primary shards / 1 replica each
B. 8 primary shards / 15 replica each

which one would perform faster?
I would guess.. indexing speed is slower on B due to many replicas.
But what about search speed...since the search query doesn't always perform on the single server...

Cheers

9

As the number of primary shards is the same here I would guess A would be faster as it has less data meaning most of it may be cached in memory.

If the data set is small enough to be cached in both scenarios I would also test a scenario C with 1 primary shard and 15 replica shards.

It is always a balance between query latency and query throughput (number of concurrent queries that can be served) and you likely need to benchmark to find out what is best for your particular use case.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.