How does Elasticsearch process a query?

[Sourav_Paul]

I want to have understanding on how search is being performed at Elasticsearch, when we place a query?

I know, this is elaborative, but would appreciate if any expert shares deep insight on query processing.

FYI, setup has 1 node, 10 indices and 10 shards in each index.

[dadoonet (David Pilato)]

I'd probably first read https://www.elastic.co/guide/en/elasticsearch/guide/current/distributed-search.html. It's old but I think that most of the content is still accurate.

[Sourav_Paul]

Thanks David.
This link is indeed helpful. So here is my understanding after the read and please correct me, if I am wrong. I also have few questions. Would be helpful, if you can share your views on those.

1. When a query arrives, it lands on a node (this node becomes coordinating node for that request).
   Question 1 : If we have multiple nodes, do we have any Load Balancer component in ES stack, which decides which node is ideal to take the request?
2. Coordinating node creates an empty priority queue and then forwards the request to every shard in the index.
   Question 2 : If we have multiple indices, does coordinating node forwards the request to all indices parallelly or sequentially?
3. Each shard executes the query locally and adds the results in locally created priority queue.
   Question 3: What if multiple queries land on a specific shard, does it create individual queue for each request? How many such parallel queries can be handled by a specific shard?
4. Each shard returns the results to coordinating node, which merges these results into it's own priority queue to produce final set of results.
   Question 4 : When we have multiple requests coming into ES, does it maintain any queue for these incoming requests?
   Question 5 : How many parallel threads at the node level and as a whole, ES can spawn to cater these incoming requests? Does ES have any limit?

[dadoonet (David Pilato)]

Question 1 : If we have multiple nodes, do we have any Load Balancer component in ES stack, which decides which node is ideal to take the request?

No. It just goes to the node you connected to.

Coordinating node creates an empty priority queue and then forwards the request to every shard in the index.

I'd say not to every shard but one copy of each shard of the index.

Question 2 : If we have multiple indices, does coordinating node forwards the request to all indices parallelly or sequentially?

In parallel.

Question 3 : What if multiple queries land on a specific shard, does it create individual queue for each request? How many such parallel queries can be handled by a specific shard?

Yes and I don't know.

Question 4 : When we have multiple requests coming into ES, does it maintain any queue for these incoming requests?

Yes.

Question 5 : How many parallel threads at the node level and as a whole, ES can spawn to cater these incoming requests? Does ES have any limit?

I don't really know and yes there are limits. Have a look at Thread pools | Elasticsearch Guide [8.11] | Elastic

[Sourav_Paul]

Thanks again David for answering my questions. I have few more doubts and will be great, if you can answer them as well when you have time.

• When we have a cluster with multiple nodes, can we have common end point for searches and implement it through Reverse Proxy (like Nginx) - which can eventually be used as Load balancer and ensure less loaded node gets the search request, instead of randomly picked node.

• If you are aware, can you please provide any link or pointers, on how many parallel queries can each shard execute?

• Thanks for sharing the link on thread module.

[DavidTurner]

@Sourav_Paul

If you are aware, can you please provide any link or pointers, on how many parallel queries can each shard execute?

There is no simple answer to this question. Searches run on the search or search_throttled threadpools on each node, but run in multiple phases. They may also contend for resources other than CPU (e.g. heap space or filesystem cache or IO bandwidth) resulting in worse overall performance at higher levels of parallelism. The only way to find a useful answer is by carefully benchmarking your specific system with your specific data and workload.

[Sourav_Paul]

Thanks David. While I do agree with you, to find right set of configurable setting, we need benchmarking with specified data set and workload. My question was more on available configurable settings at the shard level with default values and whether they are tunable or not.