Asymmetric server load in a 2 node 1-replica cluster

Elastic Stack Elasticsearch
1

hi,
i have a 2 node ES cluster with identical hosts.. the only difference is
that they are in different AZs in aws and one is using ebs instead of
ephemeral store. i have a replica setting of 1 (so 2 copies). the problem
im seeing is that the ebs based node is pegging the cpu.. there is nothing
else running on that host. bigdesk indicates that there are 137k file
descriptors on this one vs 3k on the ephemeral one. number of threads in
hot host is more than twice of the cold one. used heap is also higher in
hot host. they both are started with same max heap of 8gb.

any ideas on what could be going on?

thanks

--

2

If you're running searches, are you setting any shard preference options on
the search side?
http://www.elasticsearch.org/guide/reference/api/search/preference.html

If you're setting primary or primary_first that could cause
it. Alternatively, if you're setting local, but requests aren't round
robined, that could also account for it.

The file descriptor count is very odd. I have no explanation for what is
causing that. I'd recommend moving to a common config to see if that sheds
any light on things.

Best Regards,
Paul

On Tuesday, October 16, 2012 9:16:09 AM UTC-6, T Vinod Gupta wrote:

hi,
i have a 2 node ES cluster with identical hosts.. the only difference is
that they are in different AZs in aws and one is using ebs instead of
ephemeral store. i have a replica setting of 1 (so 2 copies). the problem
im seeing is that the ebs based node is pegging the cpu.. there is nothing
else running on that host. bigdesk indicates that there are 137k file
descriptors on this one vs 3k on the ephemeral one. number of threads in
hot host is more than twice of the cold one. used heap is also higher in
hot host. they both are started with same max heap of 8gb.

any ideas on what could be going on?

thanks

--

3

im not setting any shard preferences.. both configs are identical.
also, i have a load balancer node in front of these 2 hosts. the LB is
configured with no-data setting. all searches and index calls go through
the LB. nobody calls the hosts behind it directly.

does that change your answer?

thanks

On Tue, Oct 16, 2012 at 10:43 AM, ppearcy ppearcy@gmail.com wrote:

If you're running searches, are you setting any shard preference options
on the search side?
http://www.elasticsearch.org/guide/reference/api/search/preference.html

If you're setting primary or primary_first that could cause
it. Alternatively, if you're setting local, but requests aren't round
robined, that could also account for it.

The file descriptor count is very odd. I have no explanation for what is
causing that. I'd recommend moving to a common config to see if that sheds
any light on things.

Best Regards,
Paul

On Tuesday, October 16, 2012 9:16:09 AM UTC-6, T Vinod Gupta wrote:

hi,
i have a 2 node ES cluster with identical hosts.. the only difference is
that they are in different AZs in aws and one is using ebs instead of
ephemeral store. i have a replica setting of 1 (so 2 copies). the problem
im seeing is that the ebs based node is pegging the cpu.. there is nothing
else running on that host. bigdesk indicates that there are 137k file
descriptors on this one vs 3k on the ephemeral one. number of threads in
hot host is more than twice of the cold one. used heap is also higher in
hot host. they both are started with same max heap of 8gb.

any ideas on what could be going on?

thanks

--

--

4