AWS Spot + Reserved Instances

Elastic Stack Elasticsearch
1

I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed
reserved (reliable) instances. I'm now exploring the possibility of
introducing '3Y' AWS spot instances (unreliable but very cheap) with the
same specs into the system, turning them on and off depending on the load
on the cluster.

So:
'Y' reserved (reliable) machines
'3Y' spot (unreliable) machines

I want at least one copy of each shard to be available on the 'Y' reserved
(reliable) machines, therefore I've placed all 'Y' reserved instances in
"rack1" and '3Y' spot instances in "rack2" with forced awareness
(http://www.elasticsearch.org/guide/reference/modules/cluster.html) and one
replica for each shard.

Thus, the Y reserved instances will now have ~3 times as many shards as the
3Y spot instances and "rack1" will cumulatively have lesser memory and CPU.
As a result, will searches that are directed to the 'Y' reserved instances
be slower than searches that are directed to the '3Y' spot instances? Or
will a search directed at "rack1" (which, within itself, contains 1 copy of
each shard) be scattered/gathered across all nodes, including those in
"rack2"?

--

2

Hello Govind,

On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar
govind201@gmail.comwrote:

I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed
reserved (reliable) instances. I'm now exploring the possibility of
introducing '3Y' AWS spot instances (unreliable but very cheap) with the
same specs into the system, turning them on and off depending on the load
on the cluster.

So:
'Y' reserved (reliable) machines
'3Y' spot (unreliable) machines

I want at least one copy of each shard to be available on the 'Y' reserved
(reliable) machines, therefore I've placed all 'Y' reserved instances in
"rack1" and '3Y' spot instances in "rack2" with forced awareness (
Elasticsearch Platform — Find real-time answers at scale | Elastic) and
one replica for each shard.

Thus, the Y reserved instances will now have ~3 times as many shards as
the 3Y spot instances and "rack1" will cumulatively have lesser memory and
CPU. As a result, will searches that are directed to the 'Y' reserved
instances be slower than searches that are directed to the '3Y' spot
instances?

Yes, searches that will hit the busy nodes should be slower. But you can
fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4
as equal numbers within your poll of 3Y spot instances. Then make 3
replicas instead of 1 and you should have everything balanced. Plus, your
cluster should handle more concurrent searches.

Or will a search directed at "rack1" (which, within itself, contains 1
copy of each shard) be scattered/gathered across all nodes, including those
in "rack2"?

AFAIK, ES simply round robins between the needed shards (primaries and
replicas don't matter here, unless you specify that in your search
preferences[0]). So it won't be aware that some shard will return results
faster than others.

[0] Elasticsearch Platform — Find real-time answers at scale | Elastic

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

3

Hi Radu,

Thanks for that. Are there any good resources for better understanding how
the query scattering and gathering process is conducted? If I were to
create 3 spot racks + 1 reserved rack as you mentioned, will average query
response time improve (aside from the benefit of support for more
concurrent connections + larger cache) , i.e., will the query be scattered
across all the racks, making use of CPU resources in all racks?

Also, are the caches of each of these nodes interlinked - if a filter is
cached in node A but the query is directed to node B, will the cached
filter on node A be used or will the filter be rebuilt on node B?

Govind

On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote:

Hello Govind,

On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar <govi...@gmail.com<javascript:>

wrote:

I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed
reserved (reliable) instances. I'm now exploring the possibility of
introducing '3Y' AWS spot instances (unreliable but very cheap) with the
same specs into the system, turning them on and off depending on the load
on the cluster.

So:
'Y' reserved (reliable) machines
'3Y' spot (unreliable) machines

I want at least one copy of each shard to be available on the 'Y'
reserved (reliable) machines, therefore I've placed all 'Y' reserved
instances in "rack1" and '3Y' spot instances in "rack2" with forced
awareness (
Elasticsearch Platform — Find real-time answers at scale | Elastic) and
one replica for each shard.

Thus, the Y reserved instances will now have ~3 times as many shards as
the 3Y spot instances and "rack1" will cumulatively have lesser memory and
CPU. As a result, will searches that are directed to the 'Y' reserved
instances be slower than searches that are directed to the '3Y' spot
instances?

Yes, searches that will hit the busy nodes should be slower. But you can
fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4
as equal numbers within your poll of 3Y spot instances. Then make 3
replicas instead of 1 and you should have everything balanced. Plus, your
cluster should handle more concurrent searches.

Or will a search directed at "rack1" (which, within itself, contains 1
copy of each shard) be scattered/gathered across all nodes, including those
in "rack2"?

AFAIK, ES simply round robins between the needed shards (primaries and
replicas don't matter here, unless you specify that in your search
preferences[0]). So it won't be aware that some shard will return results
faster than others.

[0]
Elasticsearch Platform — Find real-time answers at scale | Elastic

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

4

Hi Govind,

On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar govind201@gmail.comwrote:

Hi Radu,

Thanks for that. Are there any good resources for better understanding how
the query scattering and gathering process is conducted?

I think this is a good place to look:

If I were to create 3 spot racks + 1 reserved rack as you mentioned, will
average query response time improve (aside from the benefit of support for
more concurrent connections + larger cache) , i.e., will the query be
scattered across all the racks, making use of CPU resources in all racks?

One query is sent to one set of shards (whether they are primaries or
replicas it doesn't matter). So in the 4-rack scenario, if you run a single
query when all is quiet and we ignore the state of caches and other
possible variables, you should get the same X response time no matter which
shards your query hits. In your 2-rack scenario, I would assume queries
that hit shards with 3x nodes would be faster. So the average query time
should be better.

However, if you have more than 2 concurrent queries on average, I would
expect the 4-rack scenario to get you better average times. And more
consistent times of course - because all nodes would have the same load.
Plus there's less chance of a bottleneck on your reserved instances.

I'd also recommend to run a test and see the real-life results. And observe
the load, too.

Also, are the caches of each of these nodes interlinked - if a filter is

cached in node A but the query is directed to node B, will the cached
filter on node A be used or will the filter be rebuilt on node B?

The filter will run on the selected shard and benefit from the cache of the
node where the shard is located. So node B won't benefit from the cache on
node A.

You can warm up the caches if you use 0.20+ by using the Warmers API:

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

Govind

On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote:

Hello Govind,

On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar govi...@gmail.comwrote:

I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed
reserved (reliable) instances. I'm now exploring the possibility of
introducing '3Y' AWS spot instances (unreliable but very cheap) with the
same specs into the system, turning them on and off depending on the load
on the cluster.

So:
'Y' reserved (reliable) machines
'3Y' spot (unreliable) machines

I want at least one copy of each shard to be available on the 'Y'
reserved (reliable) machines, therefore I've placed all 'Y' reserved
instances in "rack1" and '3Y' spot instances in "rack2" with forced
awareness (Elasticsearch Platform — Find real-time answers at scale | Elastic**
cluster.htmlhttp://www.elasticsearch.org/guide/reference/modules/cluster.html)
and one replica for each shard.

Thus, the Y reserved instances will now have ~3 times as many shards as
the 3Y spot instances and "rack1" will cumulatively have lesser memory and
CPU. As a result, will searches that are directed to the 'Y' reserved
instances be slower than searches that are directed to the '3Y' spot
instances?

Yes, searches that will hit the busy nodes should be slower. But you can
fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4
as equal numbers within your poll of 3Y spot instances. Then make 3
replicas instead of 1 and you should have everything balanced. Plus, your
cluster should handle more concurrent searches.

Or will a search directed at "rack1" (which, within itself, contains 1
copy of each shard) be scattered/gathered across all nodes, including those
in "rack2"?

AFAIK, ES simply round robins between the needed shards (primaries and
replicas don't matter here, unless you specify that in your search
preferences[0]). So it won't be aware that some shard will return results
faster than others.

[0] Elasticsearch Platform — Find real-time answers at scale | Elastic**
preference.htmlhttp://www.elasticsearch.org/guide/reference/api/search/preference.html

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

--

5

Just an important note here, when using awareness allocation, and lets say you "hit" a node on rack1 (i.e. send an HTTP search request that will hit nodeX on rack1), it will prefer to execute the search "within the rack" if possible, i.e., it will prefer copies of shards that exist on nodes within rack1, compared to going "outside" of the rack.

On Jan 5, 2013, at 5:08 PM, Radu Gheorghe radu.gheorghe@sematext.com wrote:

Hi Govind,

On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar govind201@gmail.com wrote:
Hi Radu,

Thanks for that. Are there any good resources for better understanding how the query scattering and gathering process is conducted?

I think this is a good place to look:
Elasticsearch Platform — Find real-time answers at scale | Elastic

If I were to create 3 spot racks + 1 reserved rack as you mentioned, will average query response time improve (aside from the benefit of support for more concurrent connections + larger cache) , i.e., will the query be scattered across all the racks, making use of CPU resources in all racks?

One query is sent to one set of shards (whether they are primaries or replicas it doesn't matter). So in the 4-rack scenario, if you run a single query when all is quiet and we ignore the state of caches and other possible variables, you should get the same X response time no matter which shards your query hits. In your 2-rack scenario, I would assume queries that hit shards with 3x nodes would be faster. So the average query time should be better.

However, if you have more than 2 concurrent queries on average, I would expect the 4-rack scenario to get you better average times. And more consistent times of course - because all nodes would have the same load. Plus there's less chance of a bottleneck on your reserved instances.

I'd also recommend to run a test and see the real-life results. And observe the load, too.

Also, are the caches of each of these nodes interlinked - if a filter is cached in node A but the query is directed to node B, will the cached filter on node A be used or will the filter be rebuilt on node B?

The filter will run on the selected shard and benefit from the cache of the node where the shard is located. So node B won't benefit from the cache on node A.

You can warm up the caches if you use 0.20+ by using the Warmers API:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

Govind

On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote:
Hello Govind,

On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar govi...@gmail.com wrote:
I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed reserved (reliable) instances. I'm now exploring the possibility of introducing '3Y' AWS spot instances (unreliable but very cheap) with the same specs into the system, turning them on and off depending on the load on the cluster.

So:
'Y' reserved (reliable) machines
'3Y' spot (unreliable) machines

I want at least one copy of each shard to be available on the 'Y' reserved (reliable) machines, therefore I've placed all 'Y' reserved instances in "rack1" and '3Y' spot instances in "rack2" with forced awareness (Elasticsearch Platform — Find real-time answers at scale | Elastic) and one replica for each shard.

Thus, the Y reserved instances will now have ~3 times as many shards as the 3Y spot instances and "rack1" will cumulatively have lesser memory and CPU. As a result, will searches that are directed to the 'Y' reserved instances be slower than searches that are directed to the '3Y' spot instances?

Yes, searches that will hit the busy nodes should be slower. But you can fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4 as equal numbers within your poll of 3Y spot instances. Then make 3 replicas instead of 1 and you should have everything balanced. Plus, your cluster should handle more concurrent searches.

Or will a search directed at "rack1" (which, within itself, contains 1 copy of each shard) be scattered/gathered across all nodes, including those in "rack2"?

AFAIK, ES simply round robins between the needed shards (primaries and replicas don't matter here, unless you specify that in your search preferences[0]). So it won't be aware that some shard will return results faster than others.

[0] Elasticsearch Platform — Find real-time answers at scale | Elastic

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

--

--

6

Thanks Radu and Shay.

I did a test setup as follows to evaluate performance:

  • 91 shards. 3 replicas of each shard (including the primary).
  • 1 node in rack 1.
  • 1 node in rack 2.
  • 2 nodes in rack 3.

Racks 1 and 2 seem fine. But the shards in rack 3 are imbalanced - one node
has 30 shards and the other has 61 shards. I thought I'd get a 45-46 split.
Am I missing something here?

On Sunday, January 6, 2013 2:42:16 PM UTC-8, kimchy wrote:

Just an important note here, when using awareness allocation, and lets say
you "hit" a node on rack1 (i.e. send an HTTP search request that will hit
nodeX on rack1), it will prefer to execute the search "within the rack" if
possible, i.e., it will prefer copies of shards that exist on nodes within
rack1, compared to going "outside" of the rack.

On Jan 5, 2013, at 5:08 PM, Radu Gheorghe <radu.g...@sematext.com<javascript:>>
wrote:

Hi Govind,

On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar <govi...@gmail.com<javascript:>

wrote:

Hi Radu,

Thanks for that. Are there any good resources for better understanding
how the query scattering and gathering process is conducted?

I think this is a good place to look:
Elasticsearch Platform — Find real-time answers at scale | Elastic

If I were to create 3 spot racks + 1 reserved rack as you mentioned, will
average query response time improve (aside from the benefit of support for
more concurrent connections + larger cache) , i.e., will the query be
scattered across all the racks, making use of CPU resources in all racks?

One query is sent to one set of shards (whether they are primaries or
replicas it doesn't matter). So in the 4-rack scenario, if you run a single
query when all is quiet and we ignore the state of caches and other
possible variables, you should get the same X response time no matter which
shards your query hits. In your 2-rack scenario, I would assume queries
that hit shards with 3x nodes would be faster. So the average query time
should be better.

However, if you have more than 2 concurrent queries on average, I would
expect the 4-rack scenario to get you better average times. And more
consistent times of course - because all nodes would have the same load.
Plus there's less chance of a bottleneck on your reserved instances.

I'd also recommend to run a test and see the real-life results. And
observe the load, too.

Also, are the caches of each of these nodes interlinked - if a filter is

cached in node A but the query is directed to node B, will the cached
filter on node A be used or will the filter be rebuilt on node B?

The filter will run on the selected shard and benefit from the cache of
the node where the shard is located. So node B won't benefit from the cache
on node A.

You can warm up the caches if you use 0.20+ by using the Warmers API:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

Govind

On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote:

Hello Govind,

On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar <govi...@gmail.com

wrote:

I currently run my Elasticsearch cluster on 'Y' number of AWS EBS
backed reserved (reliable) instances. I'm now exploring the possibility of
introducing '3Y' AWS spot instances (unreliable but very cheap) with the
same specs into the system, turning them on and off depending on the load
on the cluster.

So:
'Y' reserved (reliable) machines
'3Y' spot (unreliable) machines

I want at least one copy of each shard to be available on the 'Y'
reserved (reliable) machines, therefore I've placed all 'Y' reserved
instances in "rack1" and '3Y' spot instances in "rack2" with forced
awareness (Elasticsearch Platform — Find real-time answers at scale | Elastic**
cluster.htmlhttp://www.elasticsearch.org/guide/reference/modules/cluster.html)
and one replica for each shard.

Thus, the Y reserved instances will now have ~3 times as many shards as
the 3Y spot instances and "rack1" will cumulatively have lesser memory and
CPU. As a result, will searches that are directed to the 'Y' reserved
instances be slower than searches that are directed to the '3Y' spot
instances?

Yes, searches that will hit the busy nodes should be slower. But you can
fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4
as equal numbers within your poll of 3Y spot instances. Then make 3
replicas instead of 1 and you should have everything balanced. Plus, your
cluster should handle more concurrent searches.

Or will a search directed at "rack1" (which, within itself, contains 1
copy of each shard) be scattered/gathered across all nodes, including those
in "rack2"?

AFAIK, ES simply round robins between the needed shards (primaries and
replicas don't matter here, unless you specify that in your search
preferences[0]). So it won't be aware that some shard will return results
faster than others.

[0] Elasticsearch Platform — Find real-time answers at scale | Elastic**
preference.htmlhttp://www.elasticsearch.org/guide/reference/api/search/preference.html

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

--

--

7

Hello Govind,

That's interesting, they should have been evened out...

Can you share screenshot(s) from elasticsearch-head? Also, what version of
ES are you using?

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Mon, Jan 7, 2013 at 4:09 AM, Govind Chandrasekhar govind201@gmail.comwrote:

Thanks Radu and Shay.

I did a test setup as follows to evaluate performance:

  • 91 shards. 3 replicas of each shard (including the primary).
  • 1 node in rack 1.
  • 1 node in rack 2.
  • 2 nodes in rack 3.

Racks 1 and 2 seem fine. But the shards in rack 3 are imbalanced - one
node has 30 shards and the other has 61 shards. I thought I'd get a 45-46
split. Am I missing something here?

On Sunday, January 6, 2013 2:42:16 PM UTC-8, kimchy wrote:

Just an important note here, when using awareness allocation, and lets
say you "hit" a node on rack1 (i.e. send an HTTP search request that will
hit nodeX on rack1), it will prefer to execute the search "within the rack"
if possible, i.e., it will prefer copies of shards that exist on nodes
within rack1, compared to going "outside" of the rack.

On Jan 5, 2013, at 5:08 PM, Radu Gheorghe radu.g...@sematext.com wrote:

Hi Govind,

On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar govi...@gmail.comwrote:

Hi Radu,

Thanks for that. Are there any good resources for better understanding
how the query scattering and gathering process is conducted?

I think this is a good place to look:
Elasticsearch Platform — Find real-time answers at scale | Elastic**
search-type.htmlhttp://www.elasticsearch.org/guide/reference/api/search/search-type.html

If I were to create 3 spot racks + 1 reserved rack as you mentioned,
will average query response time improve (aside from the benefit of support
for more concurrent connections + larger cache) , i.e., will the query be
scattered across all the racks, making use of CPU resources in all racks?

One query is sent to one set of shards (whether they are primaries or
replicas it doesn't matter). So in the 4-rack scenario, if you run a single
query when all is quiet and we ignore the state of caches and other
possible variables, you should get the same X response time no matter which
shards your query hits. In your 2-rack scenario, I would assume queries
that hit shards with 3x nodes would be faster. So the average query time
should be better.

However, if you have more than 2 concurrent queries on average, I would
expect the 4-rack scenario to get you better average times. And more
consistent times of course - because all nodes would have the same load.
Plus there's less chance of a bottleneck on your reserved instances.

I'd also recommend to run a test and see the real-life results. And
observe the load, too.

Also, are the caches of each of these nodes interlinked - if a filter is

cached in node A but the query is directed to node B, will the cached
filter on node A be used or will the filter be rebuilt on node B?

The filter will run on the selected shard and benefit from the cache of
the node where the shard is located. So node B won't benefit from the cache
on node A.

You can warm up the caches if you use 0.20+ by using the Warmers API:
Elasticsearch Platform — Find real-time answers at scale | Elastic**
indices-warmers.htmlhttp://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

Govind

On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote:

Hello Govind,

On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar <
govi...@gmail.com> wrote:

I currently run my Elasticsearch cluster on 'Y' number of AWS EBS
backed reserved (reliable) instances. I'm now exploring the possibility of
introducing '3Y' AWS spot instances (unreliable but very cheap) with the
same specs into the system, turning them on and off depending on the load
on the cluster.

So:
'Y' reserved (reliable) machines
'3Y' spot (unreliable) machines

I want at least one copy of each shard to be available on the 'Y'
reserved (reliable) machines, therefore I've placed all 'Y' reserved
instances in "rack1" and '3Y' spot instances in "rack2" with forced
awareness (Elasticsearch Platform — Find real-time answers at scale | Elastic**
cluste**r.htmlhttp://www.elasticsearch.org/guide/reference/modules/cluster.html)
and one replica for each shard.

Thus, the Y reserved instances will now have ~3 times as many shards
as the 3Y spot instances and "rack1" will cumulatively have lesser memory
and CPU. As a result, will searches that are directed to the 'Y' reserved
instances be slower than searches that are directed to the '3Y' spot
instances?

Yes, searches that will hit the busy nodes should be slower. But you
can fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and
4 as equal numbers within your poll of 3Y spot instances. Then make 3
replicas instead of 1 and you should have everything balanced. Plus, your
cluster should handle more concurrent searches.

Or will a search directed at "rack1" (which, within itself, contains 1
copy of each shard) be scattered/gathered across all nodes, including those
in "rack2"?

AFAIK, ES simply round robins between the needed shards (primaries and
replicas don't matter here, unless you specify that in your search
preferences[0]). So it won't be aware that some shard will return results
faster than others.

[0] Elasticsearch Platform — Find real-time answers at scale | Elastic*
*erence.htmlhttp://www.elasticsearch.org/guide/reference/api/search/preference.html

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

--

--

--

8

Hi Radu,

Here's a screenshot of my cluster - is this what you were looking for?
"server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now
have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting
down the nodes manually and seeing if this would help - this caused a
reshuffle).

https://lh5.googleusercontent.com/-H5ygTL-4PfQ/UOulYhgnMhI/AAAAAAAAAhc/wo8WrC3nFKI/s1600/Screen+Shot+2013-01-07+at+8.48.59+PM.png

On Monday, January 7, 2013 12:09:13 AM UTC-8, Radu Gheorghe wrote:

Hello Govind,

That's interesting, they should have been evened out...

Can you share screenshot(s) from elasticsearch-head? Also, what version of
ES are you using?

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Mon, Jan 7, 2013 at 4:09 AM, Govind Chandrasekhar <govi...@gmail.com<javascript:>

wrote:

Thanks Radu and Shay.

I did a test setup as follows to evaluate performance:

  • 91 shards. 3 replicas of each shard (including the primary).
  • 1 node in rack 1.
  • 1 node in rack 2.
  • 2 nodes in rack 3.

Racks 1 and 2 seem fine. But the shards in rack 3 are imbalanced - one
node has 30 shards and the other has 61 shards. I thought I'd get a 45-46
split. Am I missing something here?

On Sunday, January 6, 2013 2:42:16 PM UTC-8, kimchy wrote:

Just an important note here, when using awareness allocation, and lets
say you "hit" a node on rack1 (i.e. send an HTTP search request that will
hit nodeX on rack1), it will prefer to execute the search "within the rack"
if possible, i.e., it will prefer copies of shards that exist on nodes
within rack1, compared to going "outside" of the rack.

On Jan 5, 2013, at 5:08 PM, Radu Gheorghe radu.g...@sematext.com
wrote:

Hi Govind,

On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar govi...@gmail.comwrote:

Hi Radu,

Thanks for that. Are there any good resources for better understanding
how the query scattering and gathering process is conducted?

I think this is a good place to look:
Elasticsearch Platform — Find real-time answers at scale | Elastic**
search-type.htmlhttp://www.elasticsearch.org/guide/reference/api/search/search-type.html

If I were to create 3 spot racks + 1 reserved rack as you mentioned,
will average query response time improve (aside from the benefit of support
for more concurrent connections + larger cache) , i.e., will the query be
scattered across all the racks, making use of CPU resources in all racks?

One query is sent to one set of shards (whether they are primaries or
replicas it doesn't matter). So in the 4-rack scenario, if you run a single
query when all is quiet and we ignore the state of caches and other
possible variables, you should get the same X response time no matter which
shards your query hits. In your 2-rack scenario, I would assume queries
that hit shards with 3x nodes would be faster. So the average query time
should be better.

However, if you have more than 2 concurrent queries on average, I would
expect the 4-rack scenario to get you better average times. And more
consistent times of course - because all nodes would have the same load.
Plus there's less chance of a bottleneck on your reserved instances.

I'd also recommend to run a test and see the real-life results. And
observe the load, too.

Also, are the caches of each of these nodes interlinked - if a filter is

cached in node A but the query is directed to node B, will the cached
filter on node A be used or will the filter be rebuilt on node B?

The filter will run on the selected shard and benefit from the cache of
the node where the shard is located. So node B won't benefit from the cache
on node A.

You can warm up the caches if you use 0.20+ by using the Warmers API:
Elasticsearch Platform — Find real-time answers at scale | Elastic**
indices-warmers.htmlhttp://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

Govind

On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote:

Hello Govind,

On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar <
govi...@gmail.com> wrote:

I currently run my Elasticsearch cluster on 'Y' number of AWS EBS
backed reserved (reliable) instances. I'm now exploring the possibility of
introducing '3Y' AWS spot instances (unreliable but very cheap) with the
same specs into the system, turning them on and off depending on the load
on the cluster.

So:
'Y' reserved (reliable) machines
'3Y' spot (unreliable) machines

I want at least one copy of each shard to be available on the 'Y'
reserved (reliable) machines, therefore I've placed all 'Y' reserved
instances in "rack1" and '3Y' spot instances in "rack2" with forced
awareness (Elasticsearch Platform — Find real-time answers at scale | Elastic*
*cluste**r.htmlhttp://www.elasticsearch.org/guide/reference/modules/cluster.html)
and one replica for each shard.

Thus, the Y reserved instances will now have ~3 times as many shards
as the 3Y spot instances and "rack1" will cumulatively have lesser memory
and CPU. As a result, will searches that are directed to the 'Y' reserved
instances be slower than searches that are directed to the '3Y' spot
instances?

Yes, searches that will hit the busy nodes should be slower. But you
can fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and
4 as equal numbers within your poll of 3Y spot instances. Then make 3
replicas instead of 1 and you should have everything balanced. Plus, your
cluster should handle more concurrent searches.

Or will a search directed at "rack1" (which, within itself, contains
1 copy of each shard) be scattered/gathered across all nodes, including
those in "rack2"?

AFAIK, ES simply round robins between the needed shards (primaries and
replicas don't matter here, unless you specify that in your search
preferences[0]). So it won't be aware that some shard will return results
faster than others.

[0] Elasticsearch Platform — Find real-time answers at scale | Elastic
**erence.htmlhttp://www.elasticsearch.org/guide/reference/api/search/preference.html

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

--

--

--

9

Oh, and I failed to mention - I'm using 0.19.8

On Monday, January 7, 2013 8:54:35 PM UTC-8, Govind Chandrasekhar wrote:

Hi Radu,

Here's a screenshot of my cluster - is this what you were looking for?
"server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now
have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting
down the nodes manually and seeing if this would help - this caused a
reshuffle).

https://lh5.googleusercontent.com/-H5ygTL-4PfQ/UOulYhgnMhI/AAAAAAAAAhc/wo8WrC3nFKI/s1600/Screen+Shot+2013-01-07+at+8.48.59+PM.png

On Monday, January 7, 2013 12:09:13 AM UTC-8, Radu Gheorghe wrote:

Hello Govind,

That's interesting, they should have been evened out...

Can you share screenshot(s) from elasticsearch-head? Also, what version
of ES are you using?

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

On Mon, Jan 7, 2013 at 4:09 AM, Govind Chandrasekhar govi...@gmail.comwrote:

Thanks Radu and Shay.

I did a test setup as follows to evaluate performance:

  • 91 shards. 3 replicas of each shard (including the primary).
  • 1 node in rack 1.
  • 1 node in rack 2.
  • 2 nodes in rack 3.

Racks 1 and 2 seem fine. But the shards in rack 3 are imbalanced - one
node has 30 shards and the other has 61 shards. I thought I'd get a 45-46
split. Am I missing something here?

On Sunday, January 6, 2013 2:42:16 PM UTC-8, kimchy wrote:

Just an important note here, when using awareness allocation, and lets
say you "hit" a node on rack1 (i.e. send an HTTP search request that will
hit nodeX on rack1), it will prefer to execute the search "within the rack"
if possible, i.e., it will prefer copies of shards that exist on nodes
within rack1, compared to going "outside" of the rack.

On Jan 5, 2013, at 5:08 PM, Radu Gheorghe radu.g...@sematext.com
wrote:

Hi Govind,

On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar <govi...@gmail.com

wrote:

Hi Radu,

Thanks for that. Are there any good resources for better understanding
how the query scattering and gathering process is conducted?

I think this is a good place to look:
Elasticsearch Platform — Find real-time answers at scale | Elastic**
search-type.htmlhttp://www.elasticsearch.org/guide/reference/api/search/search-type.html

If I were to create 3 spot racks + 1 reserved rack as you mentioned,
will average query response time improve (aside from the benefit of support
for more concurrent connections + larger cache) , i.e., will the query be
scattered across all the racks, making use of CPU resources in all racks?

One query is sent to one set of shards (whether they are primaries or
replicas it doesn't matter). So in the 4-rack scenario, if you run a single
query when all is quiet and we ignore the state of caches and other
possible variables, you should get the same X response time no matter which
shards your query hits. In your 2-rack scenario, I would assume queries
that hit shards with 3x nodes would be faster. So the average query time
should be better.

However, if you have more than 2 concurrent queries on average, I would
expect the 4-rack scenario to get you better average times. And more
consistent times of course - because all nodes would have the same load.
Plus there's less chance of a bottleneck on your reserved instances.

I'd also recommend to run a test and see the real-life results. And
observe the load, too.

Also, are the caches of each of these nodes interlinked - if a filter

is cached in node A but the query is directed to node B, will the cached
filter on node A be used or will the filter be rebuilt on node B?

The filter will run on the selected shard and benefit from the cache of
the node where the shard is located. So node B won't benefit from the cache
on node A.

You can warm up the caches if you use 0.20+ by using the Warmers API:
Elasticsearch Platform — Find real-time answers at scale | Elastic**
indices-warmers.htmlhttp://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

Govind

On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote:

Hello Govind,

On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar <
govi...@gmail.com> wrote:

I currently run my Elasticsearch cluster on 'Y' number of AWS EBS
backed reserved (reliable) instances. I'm now exploring the possibility of
introducing '3Y' AWS spot instances (unreliable but very cheap) with the
same specs into the system, turning them on and off depending on the load
on the cluster.

So:
'Y' reserved (reliable) machines
'3Y' spot (unreliable) machines

I want at least one copy of each shard to be available on the 'Y'
reserved (reliable) machines, therefore I've placed all 'Y' reserved
instances in "rack1" and '3Y' spot instances in "rack2" with forced
awareness (Elasticsearch Platform — Find real-time answers at scale | Elastic
cluster.htmlhttp://www.elasticsearch.org/guide/reference/modules/cluster.html)
and one replica for each shard.

Thus, the Y reserved instances will now have ~3 times as many shards
as the 3Y spot instances and "rack1" will cumulatively have lesser memory
and CPU. As a result, will searches that are directed to the 'Y' reserved
instances be slower than searches that are directed to the '3Y' spot
instances?

Yes, searches that will hit the busy nodes should be slower. But you
can fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and
4 as equal numbers within your poll of 3Y spot instances. Then make 3
replicas instead of 1 and you should have everything balanced. Plus, your
cluster should handle more concurrent searches.

Or will a search directed at "rack1" (which, within itself, contains
1 copy of each shard) be scattered/gathered across all nodes, including
those in "rack2"?

AFAIK, ES simply round robins between the needed shards (primaries
and replicas don't matter here, unless you specify that in your search
preferences[0]). So it won't be aware that some shard will return results
faster than others.

[0] Elasticsearch Platform — Find real-time answers at scale | Elastic**
pref**erence.htmlhttp://www.elasticsearch.org/guide/reference/api/search/preference.html

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

--

--

--

10

Hi Govind,

On Tue, Jan 8, 2013 at 6:54 AM, Govind Chandrasekhar govind201@gmail.comwrote:

Hi Radu,

Here's a screenshot of my cluster - is this what you were looking for?
"server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now
have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting
down the nodes manually and seeing if this would help - this caused a
reshuffle).

Actually, I was interested in a screenshot from elasticsearch-head:
http://mobz.github.com/elasticsearch-head/

So I can see better which shard on which index is allocated to which node.
I was also interested in how many indices you have and why you have so many
total shards.

While things are better now, you can check the same elasticsearch-head
plugin to see if shards are still moving around. You can also get this
information via the Cluster Health API:

If shards are inbalanced and nothing is moving, you can check out the logs
for clues.

Ultimately, you can try to move shards manually using the Cluster Reroute
API:

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

11

Hi Radu,

On Wednesday, January 9, 2013 1:13:03 AM UTC-8, Radu Gheorghe wrote:

Hi Govind,

On Tue, Jan 8, 2013 at 6:54 AM, Govind Chandrasekhar <govi...@gmail.com<javascript:>

wrote:

Hi Radu,

Here's a screenshot of my cluster - is this what you were looking for?
"server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now
have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting
down the nodes manually and seeing if this would help - this caused a
reshuffle).

Actually, I was interested in a screenshot from elasticsearch-head:
http://mobz.github.com/elasticsearch-head/

So I can see better which shard on which index is allocated to which node.
I was also interested in how many indices you have and why you have so many
total shards.

The elasticsearch-head screenshot's far too spread out because of the
number of nodes unfortunately. I have 23 main indices - one for each
product category -, with 3 shards each. The remaining indices - 22 shards -
are either negligibly small (will the presence of empty shards cause an
issue?). When a user makes a query, it is automatically directed to any one
index (3 shards), though I am eventually hoping to support queries that
span all 69 shards.

While things are better now, you can check the same elasticsearch-head
plugin to see if shards are still moving around. You can also get this
information via the Cluster Health API:
Elasticsearch Platform — Find real-time answers at scale | Elastic

If shards are inbalanced and nothing is moving, you can check out the logs
for clues.

Ultimately, you can try to move shards manually using the Cluster Reroute
API:
Elasticsearch Platform — Find real-time answers at scale | Elastic

Yup, this is what I"m resorting to!

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

12

An update:
I've tried many combinations of racks with awareness, and the shards are
usually unbalanced within machines of the same rack. I suspect that this a
bug?

Everything else works fine (I'm using the the cluster reroute API for the
moment).

On Wednesday, January 9, 2013 9:18:30 AM UTC-8, Govind Chandrasekhar wrote:

Hi Radu,

On Wednesday, January 9, 2013 1:13:03 AM UTC-8, Radu Gheorghe wrote:

Hi Govind,

On Tue, Jan 8, 2013 at 6:54 AM, Govind Chandrasekhar govi...@gmail.comwrote:

Hi Radu,

Here's a screenshot of my cluster - is this what you were looking for?
"server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now
have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting
down the nodes manually and seeing if this would help - this caused a
reshuffle).

Actually, I was interested in a screenshot from elasticsearch-head:
http://mobz.github.com/elasticsearch-head/

So I can see better which shard on which index is allocated to which
node. I was also interested in how many indices you have and why you have
so many total shards.

The elasticsearch-head screenshot's far too spread out because of the
number of nodes unfortunately. I have 23 main indices - one for each
product category -, with 3 shards each. The remaining indices - 22 shards -
are either negligibly small (will the presence of empty shards cause an
issue?). When a user makes a query, it is automatically directed to any one
index (3 shards), though I am eventually hoping to support queries that
span all 69 shards.

While things are better now, you can check the same elasticsearch-head
plugin to see if shards are still moving around. You can also get this
information via the Cluster Health API:
Elasticsearch Platform — Find real-time answers at scale | Elastic

If shards are inbalanced and nothing is moving, you can check out the
logs for clues.

Ultimately, you can try to move shards manually using the Cluster Reroute
API:

Elasticsearch Platform — Find real-time answers at scale | Elastic

Yup, this is what I"m resorting to!

Best regards,
Radu

http://sematext.com/ -- Elasticsearch -- Solr -- Lucene

--

13

is dead
search request preference

I have been looking for this link for several hours, thanks !