I have 2 different indices within an ElasticSearch cluster with 6 nodes. One index (index A) has 2 primary shards and 2 replicas per giving us 6 total shards. The other index (index B) has 5 primary shards and 1 replica per giving us 10 total shards. When I run the profiler on my query for index A i can see that the query is only being made on 2 different shards(primaries and/or replicas). When I run the profiler on the same query on index B the query is being made on 5 different shards(primaries and/or replicas). It seems that the query is routed to only x shards where x is the number of primary shards designated for that index, but both indices have at least 6 shards for the query to be made on. Is this behavior known or am I missing something?
Yes, that sounds like the expected behaviour. Each copy of a shard (primary or replica) holds the same data, so there is no point in sending the same search to multiple copies of the same shard.
If I am understanding correctly then a query will only get routed to x number of shards where x is the number of primaries. So even if your schema was 2:5 giving you 10 shards the query would only be sent to 2 shards? Does it make sense that I am seeing much better performance when making the query on index B in the example above with a 5:1 schema vs index A with a 2:2 schema.
Yes, that's correct. However if you send a large number of searches then they will be distributed evenly across all available shards, and that's the case we normally optimise for.
The situation you are describing above is when many queries are ran in parallel correct? In this case ElasticSearch will distribute the queries across all the appropriate shards for the indices being queried.
Are you also saying that the single query situation I was describing where the query is sent to multiple shards is being optimized such that the more shards queried yields faster speeds?
I guess what I am trying to understand is why the same query runs faster on an index with 5:1 schema vs an index with 2:2 schema.
Do these indices hold exactly the same data?
yes these indices hold the same data, can show screenshots of the query results if that would help/profile api results
Each query against a shard is single-threaded (queries against different shards run in parallel) which means fewer threads are effectively used for the index that is slower. The shards in the slower index should also be larger which also can impact query latency. Exactly what the difference is will depend on your data and queries.
More shards can therefore be faster if you have few concurrent queries but this can change as the number of concurrent queries increase.
It makes sense that the queries run on different shards run in parallel but why does this make the query being routed to 5 shards faster if each thread is doing the exact same thing?
You have 5 threads processing a smaller piece of data each compared to the 2 threads that need to processing considerably more data. It is a simplification but explains the difference.
Does this mean that the work for a single query would be split amongst the 5 threads? I was not aware of this behavior. This is not the same situation as what DavidTurner was describing above where he said a large number of searches will be evenly distributed across all available shards, just to clarify
Yes, a thread per shard is used and if there are not enough threads in the thread pool the jobs are queued up.
What I described happen for every query. If you have multiple concurrent queries the load is distributed across the available shards just as David described, but each is still served by multiple threads assuming they target more than one shard.
So to give an example, if we had an query/aggregation being performed on an index with a 5:1 shard schema. Different pieces of the result would be returned by each of the 5 threads used to complete the task?
Really appreciate your time and patience in explaining this to me
Yes, the results from the 5 shards would be returned to the coordinating thread which computes the final result.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.