How does non-data node choose a node for query/fetch?


(T Vinod Gupta) #1

when a non-data node receives a search request, how does it go about
choosing the nodes that have replicas of a shard? e.g if number of replicas
is 1 and there are 2 data nodes then both nodes have complete index data
and shards. if there is a non-data node fronting these 2 nodes and it gets
a search request, how does it distribute the work? is there a setting to
make it fire parallel redundant requests to both nodes and return the
results as soon as they are ready (without waiting for both nodes to
respond)?

i guess the situation im trying to address is a case where a slow node in a
cluster bottlenecks the cluster as a whole.

thanks

--


(Radu Gheorghe) #2

Hello,

As far as I know, a round robin is used to distribute searches between
replicas. So if a node is slower, queries hitting shards on that node
would normally be slower.

My understanding is that adding replicas help with concurrent queries.
If you run one query at a time, your searches should be just as fast.

Regarding the situation your trying to address, I think a solution
might be to divide your data into multiple indices - based on how
"hot" that data is. So I'd put data that is queried more frequently on
the faster nodes, using shard allocation filtering:
http://www.elasticsearch.org/guide/reference/index-modules/allocation.html

And you have to make sure that, within your application, you query
only the needed indices, not all of them.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Tue, Oct 30, 2012 at 12:48 AM, T Vinod Gupta tvinod@readypulse.com wrote:

when a non-data node receives a search request, how does it go about
choosing the nodes that have replicas of a shard? e.g if number of replicas
is 1 and there are 2 data nodes then both nodes have complete index data and
shards. if there is a non-data node fronting these 2 nodes and it gets a
search request, how does it distribute the work? is there a setting to make
it fire parallel redundant requests to both nodes and return the results as
soon as they are ready (without waiting for both nodes to respond)?

i guess the situation im trying to address is a case where a slow node in a
cluster bottlenecks the cluster as a whole.

thanks

--


(T Vinod Gupta) #3

thanks for the info..
now isnt that inefficient as it doesnt take advantage of caching.. lets say
1st search goes to node1 and it caches the filter results. same search is
fired again.. if it goes to node2, then you have lost cache advantage.
am i missing something?

thanks

On Tue, Oct 30, 2012 at 6:27 AM, Radu Gheorghe
radu.gheorghe@sematext.comwrote:

Hello,

As far as I know, a round robin is used to distribute searches between
replicas. So if a node is slower, queries hitting shards on that node
would normally be slower.

My understanding is that adding replicas help with concurrent queries.
If you run one query at a time, your searches should be just as fast.

Regarding the situation your trying to address, I think a solution
might be to divide your data into multiple indices - based on how
"hot" that data is. So I'd put data that is queried more frequently on
the faster nodes, using shard allocation filtering:
http://www.elasticsearch.org/guide/reference/index-modules/allocation.html

And you have to make sure that, within your application, you query
only the needed indices, not all of them.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Tue, Oct 30, 2012 at 12:48 AM, T Vinod Gupta tvinod@readypulse.com
wrote:

when a non-data node receives a search request, how does it go about
choosing the nodes that have replicas of a shard? e.g if number of
replicas
is 1 and there are 2 data nodes then both nodes have complete index data
and
shards. if there is a non-data node fronting these 2 nodes and it gets a
search request, how does it distribute the work? is there a setting to
make
it fire parallel redundant requests to both nodes and return the results
as
soon as they are ready (without waiting for both nodes to respond)?

i guess the situation im trying to address is a case where a slow node
in a
cluster bottlenecks the cluster as a whole.

thanks

--

--


(Radu Gheorghe) #4

Hello,

On Thu, Nov 1, 2012 at 4:03 AM, T Vinod Gupta tvinod@readypulse.com wrote:

thanks for the info..
now isnt that inefficient as it doesnt take advantage of caching.. lets say
1st search goes to node1 and it caches the filter results. same search is
fired again.. if it goes to node2, then you have lost cache advantage.
am i missing something?

Well, if you run a lot of queries, all nodes should warm up and you
shouldn't notice a difference. If it's not the case, you can use
Search Preferences to prefer running on the primary shards:
http://www.elasticsearch.org/guide/reference/api/search/preference.html

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

--


(Igor Motov) #5

You can also use "custom value" in search preference to make sure that all
requests that contain the same search stick to the same shard (not
necessary primary). However, in this case you will have to balance the load
between shards yourself.

On Thursday, November 1, 2012 3:52:14 AM UTC-4, Radu Gheorghe wrote:

Hello,

On Thu, Nov 1, 2012 at 4:03 AM, T Vinod Gupta <tvi...@readypulse.com<javascript:>>
wrote:

thanks for the info..
now isnt that inefficient as it doesnt take advantage of caching.. lets
say
1st search goes to node1 and it caches the filter results. same search
is
fired again.. if it goes to node2, then you have lost cache advantage.
am i missing something?

Well, if you run a lot of queries, all nodes should warm up and you
shouldn't notice a difference. If it's not the case, you can use
Search Preferences to prefer running on the primary shards:
http://www.elasticsearch.org/guide/reference/api/search/preference.html

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

--


(Otis Gospodnetić) #6

Hi Vinod,

Indeed, it is ineffcient and Radu and Igor already provided solutions. You
can see the inefficiency in a multi-node cluster (heh, a bit of word
redundancy never hurt anyone, I suppose) where you run the same query a
couple of times. You will see higher latency initially, but after a few
requests the latency will drop due to caches. Note that it's not only
about filter caches or any other ES-level caching, but also OS caches.

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Performance Monitoring - http://sematext.com/spm/indexhttp://sematext.com/spm/index.html

On Wednesday, October 31, 2012 10:03:44 PM UTC-4, T Vinod Gupta wrote:

thanks for the info..
now isnt that inefficient as it doesnt take advantage of caching.. lets
say 1st search goes to node1 and it caches the filter results. same search
is fired again.. if it goes to node2, then you have lost cache advantage.
am i missing something?

thanks

On Tue, Oct 30, 2012 at 6:27 AM, Radu Gheorghe <radu.g...@sematext.com<javascript:>

wrote:

Hello,

As far as I know, a round robin is used to distribute searches between
replicas. So if a node is slower, queries hitting shards on that node
would normally be slower.

My understanding is that adding replicas help with concurrent queries.
If you run one query at a time, your searches should be just as fast.

Regarding the situation your trying to address, I think a solution
might be to divide your data into multiple indices - based on how
"hot" that data is. So I'd put data that is queried more frequently on
the faster nodes, using shard allocation filtering:
http://www.elasticsearch.org/guide/reference/index-modules/allocation.html

And you have to make sure that, within your application, you query
only the needed indices, not all of them.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

On Tue, Oct 30, 2012 at 12:48 AM, T Vinod Gupta <tvi...@readypulse.com<javascript:>>
wrote:

when a non-data node receives a search request, how does it go about
choosing the nodes that have replicas of a shard? e.g if number of
replicas
is 1 and there are 2 data nodes then both nodes have complete index
data and
shards. if there is a non-data node fronting these 2 nodes and it gets a
search request, how does it distribute the work? is there a setting to
make
it fire parallel redundant requests to both nodes and return the
results as
soon as they are ready (without waiting for both nodes to respond)?

i guess the situation im trying to address is a case where a slow node
in a
cluster bottlenecks the cluster as a whole.

thanks

--

--


(system) #7