# AWS Spot + Reserved Instances **URL:** \ **Category:** Elasticsearch\ **Created:** [December 31, 2012, 4:27pm UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208 "2012-12-31T16:27:35Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![govind201](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@govind201](https://discuss.elastic.co/u/govind201)\ **Post date:** [December 31, 2012, 4:27pm UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/1 "2012-12-31T16:27:35Z")
I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed reserved (reliable) instances. I'm now exploring the possibility of introducing '3Y' AWS spot instances (unreliable but very cheap) with the same specs into the system, turning them on and off depending on the load on the cluster. So: 'Y' reserved (reliable) machines '3Y' spot (unreliable) machines I want at least one copy of each shard to be available on the 'Y' reserved (reliable) machines, therefore I've placed all 'Y' reserved instances in "rack1" and '3Y' spot instances in "rack2" with forced awareness ([http://www.elasticsearch.org/guide/reference/modules/cluster.html](http://www.elasticsearch.org/guide/reference/modules/cluster.html)) and one replica for each shard. Thus, the Y reserved instances will now have ~3 times as many shards as the 3Y spot instances and "rack1" will cumulatively have lesser memory and CPU. As a result, will searches that are directed to the 'Y' reserved instances be slower than searches that are directed to the '3Y' spot instances? Or will a search directed at "rack1" (which, within itself, contains 1 copy of each shard) be scattered/gathered across all nodes, including those in "rack2"? -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 3, 2013, 10:59am UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/2 "2013-01-03T10:59:32Z")
Hello Govind, On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar [govind201@gmail.com](mailto:govind201@gmail.com)wrote: > I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed > reserved (reliable) instances. I'm now exploring the possibility of > introducing '3Y' AWS spot instances (unreliable but very cheap) with the > same specs into the system, turning them on and off depending on the load > on the cluster. > > So: > 'Y' reserved (reliable) machines > '3Y' spot (unreliable) machines > > I want at least one copy of each shard to be available on the 'Y' reserved > (reliable) machines, therefore I've placed all 'Y' reserved instances in > "rack1" and '3Y' spot instances in "rack2" with forced awareness ( > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/modules/cluster.html)) and > one replica for each shard. > > Thus, the Y reserved instances will now have ~3 times as many shards as > the 3Y spot instances and "rack1" will cumulatively have lesser memory and > CPU. As a result, will searches that are directed to the 'Y' reserved > instances be slower than searches that are directed to the '3Y' spot > instances? Yes, searches that will hit the busy nodes should be slower. But you can fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4 as equal numbers within your poll of 3Y spot instances. Then make 3 replicas instead of 1 and you should have everything balanced. Plus, your cluster should handle more concurrent searches. > Or will a search directed at "rack1" (which, within itself, contains 1 > copy of each shard) be scattered/gathered across all nodes, including those > in "rack2"? AFAIK, ES simply round robins between the needed shards (primaries and replicas don't matter here, unless you specify that in your search preferences[0]). So it won't be aware that some shard will return results faster than others. [0] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/search/preference.html) ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![govind201](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@govind201](https://discuss.elastic.co/u/govind201)\ **Post date:** [January 5, 2013, 4:10am UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/3 "2013-01-05T04:10:26Z")
Hi Radu, Thanks for that. Are there any good resources for better understanding how the query scattering and gathering process is conducted? If I were to create 3 spot racks + 1 reserved rack as you mentioned, will average query response time improve (aside from the benefit of support for more concurrent connections + larger cache) , i.e., will the query be scattered across all the racks, making use of CPU resources in all racks? Also, are the caches of each of these nodes interlinked - if a filter is cached in node A but the query is directed to node B, will the cached filter on node A be used or will the filter be rebuilt on node B? Govind On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote: > Hello Govind, > > On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar \<[govi...@gmail.com](mailto:govi...@gmail.com)\ > > > wrote: > > > I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed > > reserved (reliable) instances. I'm now exploring the possibility of > > introducing '3Y' AWS spot instances (unreliable but very cheap) with the > > same specs into the system, turning them on and off depending on the load > > on the cluster. > > > > So: > > 'Y' reserved (reliable) machines > > '3Y' spot (unreliable) machines > > > > I want at least one copy of each shard to be available on the 'Y' > > reserved (reliable) machines, therefore I've placed all 'Y' reserved > > instances in "rack1" and '3Y' spot instances in "rack2" with forced > > awareness ( > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/cluster.html)) and > > one replica for each shard. > > > > Thus, the Y reserved instances will now have ~3 times as many shards as > > the 3Y spot instances and "rack1" will cumulatively have lesser memory and > > CPU. As a result, will searches that are directed to the 'Y' reserved > > instances be slower than searches that are directed to the '3Y' spot > > instances? > > Yes, searches that will hit the busy nodes should be slower. But you can > fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4 > as equal numbers within your poll of 3Y spot instances. Then make 3 > replicas instead of 1 and you should have everything balanced. Plus, your > cluster should handle more concurrent searches. > > > Or will a search directed at "rack1" (which, within itself, contains 1 > > copy of each shard) be scattered/gathered across all nodes, including those > > in "rack2"? > > AFAIK, ES simply round robins between the needed shards (primaries and > replicas don't matter here, unless you specify that in your search > preferences[0]). So it won't be aware that some shard will return results > faster than others. > > [0] > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/preference.html) > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 5, 2013, 4:08pm UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/4 "2013-01-05T16:08:33Z")
Hi Govind, On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar [govind201@gmail.com](mailto:govind201@gmail.com)wrote: > Hi Radu, > > Thanks for that. Are there any good resources for better understanding how > the query scattering and gathering process is conducted? I think this is a good place to look: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... > If I were to create 3 spot racks + 1 reserved rack as you mentioned, will > average query response time improve (aside from the benefit of support for > more concurrent connections + larger cache) , i.e., will the query be > scattered across all the racks, making use of CPU resources in all racks? One query is sent to one set of shards (whether they are primaries or replicas it doesn't matter). So in the 4-rack scenario, if you run a single query when all is quiet and we ignore the state of caches and other possible variables, you should get the same X response time no matter which shards your query hits. In your 2-rack scenario, I would assume queries that hit shards with 3x nodes would be faster. So the average query time should be better. However, if you have more than 2 concurrent queries on average, I would expect the 4-rack scenario to get you better average times. And more consistent times of course - because all nodes would have the same load. Plus there's less chance of a bottleneck on your reserved instances. I'd also recommend to run a test and see the real-life results. And observe the load, too. Also, are the caches of each of these nodes interlinked - if a filter is > cached in node A but the query is directed to node B, will the cached > filter on node A be used or will the filter be rebuilt on node B? The filter will run on the selected shard and benefit from the cache of the node where the shard is located. So node B won't benefit from the cache on node A. You can warm up the caches if you use 0.20+ by using the Warmers API: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > Govind > > On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote: > > > Hello Govind, > > > > On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar [govi...@gmail.com](mailto:govi...@gmail.com)wrote: > > > > > I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed > > > reserved (reliable) instances. I'm now exploring the possibility of > > > introducing '3Y' AWS spot instances (unreliable but very cheap) with the > > > same specs into the system, turning them on and off depending on the load > > > on the cluster. > > > > > > So: > > > 'Y' reserved (reliable) machines > > > '3Y' spot (unreliable) machines > > > > > > I want at least one copy of each shard to be available on the 'Y' > > > reserved (reliable) machines, therefore I've placed all 'Y' reserved > > > instances in "rack1" and '3Y' spot instances in "rack2" with forced > > > awareness ([Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/**guide/reference/modules/)\*\* > > > cluster.html[http://www.elasticsearch.org/guide/reference/modules/cluster.html](http://www.elasticsearch.org/guide/reference/modules/cluster.html)) > > > and one replica for each shard. > > > > > > Thus, the Y reserved instances will now have ~3 times as many shards as > > > the 3Y spot instances and "rack1" will cumulatively have lesser memory and > > > CPU. As a result, will searches that are directed to the 'Y' reserved > > > instances be slower than searches that are directed to the '3Y' spot > > > instances? > > > > Yes, searches that will hit the busy nodes should be slower. But you can > > fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4 > > as equal numbers within your poll of 3Y spot instances. Then make 3 > > replicas instead of 1 and you should have everything balanced. Plus, your > > cluster should handle more concurrent searches. > > > > > Or will a search directed at "rack1" (which, within itself, contains 1 > > > copy of each shard) be scattered/gathered across all nodes, including those > > > in "rack2"? > > > > AFAIK, ES simply round robins between the needed shards (primaries and > > replicas don't matter here, unless you specify that in your search > > preferences[0]). So it won't be aware that some shard will return results > > faster than others. > > > > [0] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/**guide/reference/api/search/)\*\* > > preference.html[http://www.elasticsearch.org/guide/reference/api/search/preference.html](http://www.elasticsearch.org/guide/reference/api/search/preference.html) > > > > ## Best regards, Radu > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > -- -- ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [January 6, 2013, 10:42pm UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/5 "2013-01-06T22:42:16Z")
Just an important note here, when using awareness allocation, and lets say you "hit" a node on rack1 (i.e. send an HTTP search request that will hit nodeX on rack1), it will prefer to execute the search "within the rack" if possible, i.e., it will prefer copies of shards that exist on nodes within rack1, compared to going "outside" of the rack. On Jan 5, 2013, at 5:08 PM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) wrote: > Hi Govind, > > On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar [govind201@gmail.com](mailto:govind201@gmail.com) wrote: > Hi Radu, > > Thanks for that. Are there any good resources for better understanding how the query scattering and gathering process is conducted? > > I think this is a good place to look: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html) > > If I were to create 3 spot racks + 1 reserved rack as you mentioned, will average query response time improve (aside from the benefit of support for more concurrent connections + larger cache) , i.e., will the query be scattered across all the racks, making use of CPU resources in all racks? > > One query is sent to one set of shards (whether they are primaries or replicas it doesn't matter). So in the 4-rack scenario, if you run a single query when all is quiet and we ignore the state of caches and other possible variables, you should get the same X response time no matter which shards your query hits. In your 2-rack scenario, I would assume queries that hit shards with 3x nodes would be faster. So the average query time should be better. > > However, if you have more than 2 concurrent queries on average, I would expect the 4-rack scenario to get you better average times. And more consistent times of course - because all nodes would have the same load. Plus there's less chance of a bottleneck on your reserved instances. > > I'd also recommend to run a test and see the real-life results. And observe the load, too. > > Also, are the caches of each of these nodes interlinked - if a filter is cached in node A but the query is directed to node B, will the cached filter on node A be used or will the filter be rebuilt on node B? > > The filter will run on the selected shard and benefit from the cache of the node where the shard is located. So node B won't benefit from the cache on node A. > > You can warm up the caches if you use 0.20+ by using the Warmers API: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html) > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > Govind > > On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote: > Hello Govind, > > On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar [govi...@gmail.com](mailto:govi...@gmail.com) wrote: > I currently run my Elasticsearch cluster on 'Y' number of AWS EBS backed reserved (reliable) instances. I'm now exploring the possibility of introducing '3Y' AWS spot instances (unreliable but very cheap) with the same specs into the system, turning them on and off depending on the load on the cluster. > > So: > 'Y' reserved (reliable) machines > '3Y' spot (unreliable) machines > > I want at least one copy of each shard to be available on the 'Y' reserved (reliable) machines, therefore I've placed all 'Y' reserved instances in "rack1" and '3Y' spot instances in "rack2" with forced awareness ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/modules/cluster.html)) and one replica for each shard. > > Thus, the Y reserved instances will now have ~3 times as many shards as the 3Y spot instances and "rack1" will cumulatively have lesser memory and CPU. As a result, will searches that are directed to the 'Y' reserved instances be slower than searches that are directed to the '3Y' spot instances? > > Yes, searches that will hit the busy nodes should be slower. But you can fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4 as equal numbers within your poll of 3Y spot instances. Then make 3 replicas instead of 1 and you should have everything balanced. Plus, your cluster should handle more concurrent searches. > > Or will a search directed at "rack1" (which, within itself, contains 1 copy of each shard) be scattered/gathered across all nodes, including those in "rack2"? > > AFAIK, ES simply round robins between the needed shards (primaries and replicas don't matter here, unless you specify that in your search preferences[0]). So it won't be aware that some shard will return results faster than others. > > [0] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/preference.html) > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > -- > > -- -- ---
**Author:** ![govind201](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@govind201](https://discuss.elastic.co/u/govind201)\ **Post date:** [January 7, 2013, 2:09am UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/6 "2013-01-07T02:09:23Z")
Thanks Radu and Shay. I did a test setup as follows to evaluate performance: - 91 shards. 3 replicas of each shard (including the primary). - 1 node in rack 1. - 1 node in rack 2. - 2 nodes in rack 3. Racks 1 and 2 seem fine. But the shards in rack 3 are imbalanced - one node has 30 shards and the other has 61 shards. I thought I'd get a 45-46 split. Am I missing something here? On Sunday, January 6, 2013 2:42:16 PM UTC-8, kimchy wrote: > Just an important note here, when using awareness allocation, and lets say > you "hit" a node on rack1 (i.e. send an HTTP search request that will hit > nodeX on rack1), it will prefer to execute the search "within the rack" if > possible, i.e., it will prefer copies of shards that exist on nodes within > rack1, compared to going "outside" of the rack. > > On Jan 5, 2013, at 5:08 PM, Radu Gheorghe \<[radu.g...@sematext.com](mailto:radu.g...@sematext.com)\\> > wrote: > > Hi Govind, > > On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar \<[govi...@gmail.com](mailto:govi...@gmail.com)\ > > > wrote: > > > Hi Radu, > > > > Thanks for that. Are there any good resources for better understanding > > how the query scattering and gathering process is conducted? > > I think this is a good place to look: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/search-type.html) > > > If I were to create 3 spot racks + 1 reserved rack as you mentioned, will > > average query response time improve (aside from the benefit of support for > > more concurrent connections + larger cache) , i.e., will the query be > > scattered across all the racks, making use of CPU resources in all racks? > > One query is sent to one set of shards (whether they are primaries or > replicas it doesn't matter). So in the 4-rack scenario, if you run a single > query when all is quiet and we ignore the state of caches and other > possible variables, you should get the same X response time no matter which > shards your query hits. In your 2-rack scenario, I would assume queries > that hit shards with 3x nodes would be faster. So the average query time > should be better. > > However, if you have more than 2 concurrent queries on average, I would > expect the 4-rack scenario to get you better average times. And more > consistent times of course - because all nodes would have the same load. > Plus there's less chance of a bottleneck on your reserved instances. > > I'd also recommend to run a test and see the real-life results. And > observe the load, too. > > Also, are the caches of each of these nodes interlinked - if a filter is > > > cached in node A but the query is directed to node B, will the cached > > filter on node A be used or will the filter be rebuilt on node B? > > The filter will run on the selected shard and benefit from the cache of > the node where the shard is located. So node B won't benefit from the cache > on node A. > > You can warm up the caches if you use 0.20+ by using the Warmers API: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html) > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > Govind > > > > On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote: > > > > > Hello Govind, > > > > > > On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar \<[govi...@gmail.com](mailto:govi...@gmail.com) > > > > > > > wrote: > > > > > > > I currently run my Elasticsearch cluster on 'Y' number of AWS EBS > > > > backed reserved (reliable) instances. I'm now exploring the possibility of > > > > introducing '3Y' AWS spot instances (unreliable but very cheap) with the > > > > same specs into the system, turning them on and off depending on the load > > > > on the cluster. > > > > > > > > So: > > > > 'Y' reserved (reliable) machines > > > > '3Y' spot (unreliable) machines > > > > > > > > I want at least one copy of each shard to be available on the 'Y' > > > > reserved (reliable) machines, therefore I've placed all 'Y' reserved > > > > instances in "rack1" and '3Y' spot instances in "rack2" with forced > > > > awareness ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/modules/)\*\* > > > > cluster.html[http://www.elasticsearch.org/guide/reference/modules/cluster.html](http://www.elasticsearch.org/guide/reference/modules/cluster.html)) > > > > and one replica for each shard. > > > > > > > > Thus, the Y reserved instances will now have ~3 times as many shards as > > > > the 3Y spot instances and "rack1" will cumulatively have lesser memory and > > > > CPU. As a result, will searches that are directed to the 'Y' reserved > > > > instances be slower than searches that are directed to the '3Y' spot > > > > instances? > > > > > > Yes, searches that will hit the busy nodes should be slower. But you can > > > fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and 4 > > > as equal numbers within your poll of 3Y spot instances. Then make 3 > > > replicas instead of 1 and you should have everything balanced. Plus, your > > > cluster should handle more concurrent searches. > > > > > > > Or will a search directed at "rack1" (which, within itself, contains 1 > > > > copy of each shard) be scattered/gathered across all nodes, including those > > > > in "rack2"? > > > > > > AFAIK, ES simply round robins between the needed shards (primaries and > > > replicas don't matter here, unless you specify that in your search > > > preferences[0]). So it won't be aware that some shard will return results > > > faster than others. > > > > > > [0] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/api/search/)\*\* > > > preference.html[http://www.elasticsearch.org/guide/reference/api/search/preference.html](http://www.elasticsearch.org/guide/reference/api/search/preference.html) > > > > > > ## Best regards, Radu > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > -- > > -- -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 7, 2013, 8:09am UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/7 "2013-01-07T08:09:13Z")
Hello Govind, That's interesting, they should have been evened out... Can you share screenshot(s) from elasticsearch-head? Also, what version of ES are you using? ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene On Mon, Jan 7, 2013 at 4:09 AM, Govind Chandrasekhar [govind201@gmail.com](mailto:govind201@gmail.com)wrote: > Thanks Radu and Shay. > > I did a test setup as follows to evaluate performance: > > - 91 shards. 3 replicas of each shard (including the primary). > - 1 node in rack 1. > - 1 node in rack 2. > - 2 nodes in rack 3. > > Racks 1 and 2 seem fine. But the shards in rack 3 are imbalanced - one > node has 30 shards and the other has 61 shards. I thought I'd get a 45-46 > split. Am I missing something here? > > On Sunday, January 6, 2013 2:42:16 PM UTC-8, kimchy wrote: > > > Just an important note here, when using awareness allocation, and lets > > say you "hit" a node on rack1 (i.e. send an HTTP search request that will > > hit nodeX on rack1), it will prefer to execute the search "within the rack" > > if possible, i.e., it will prefer copies of shards that exist on nodes > > within rack1, compared to going "outside" of the rack. > > > > On Jan 5, 2013, at 5:08 PM, Radu Gheorghe [radu.g...@sematext.com](mailto:radu.g...@sematext.com) wrote: > > > > Hi Govind, > > > > On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar [govi...@gmail.com](mailto:govi...@gmail.com)wrote: > > > > > Hi Radu, > > > > > > Thanks for that. Are there any good resources for better understanding > > > how the query scattering and gathering process is conducted? > > > > I think this is a good place to look: > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/**guide/reference/api/search/)\*\* > > search-type.html[http://www.elasticsearch.org/guide/reference/api/search/search-type.html](http://www.elasticsearch.org/guide/reference/api/search/search-type.html) > > > > > If I were to create 3 spot racks + 1 reserved rack as you mentioned, > > > will average query response time improve (aside from the benefit of support > > > for more concurrent connections + larger cache) , i.e., will the query be > > > scattered across all the racks, making use of CPU resources in all racks? > > > > One query is sent to one set of shards (whether they are primaries or > > replicas it doesn't matter). So in the 4-rack scenario, if you run a single > > query when all is quiet and we ignore the state of caches and other > > possible variables, you should get the same X response time no matter which > > shards your query hits. In your 2-rack scenario, I would assume queries > > that hit shards with 3x nodes would be faster. So the average query time > > should be better. > > > > However, if you have more than 2 concurrent queries on average, I would > > expect the 4-rack scenario to get you better average times. And more > > consistent times of course - because all nodes would have the same load. > > Plus there's less chance of a bottleneck on your reserved instances. > > > > I'd also recommend to run a test and see the real-life results. And > > observe the load, too. > > > > Also, are the caches of each of these nodes interlinked - if a filter is > > > > > cached in node A but the query is directed to node B, will the cached > > > filter on node A be used or will the filter be rebuilt on node B? > > > > The filter will run on the selected shard and benefit from the cache of > > the node where the shard is located. So node B won't benefit from the cache > > on node A. > > > > You can warm up the caches if you use 0.20+ by using the Warmers API: > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/**guide/reference/api/admin-)\*\* > > indices-warmers.html[http://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html) > > > > ## Best regards, Radu > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > Govind > > > > > > On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote: > > > > > > > Hello Govind, > > > > > > > > On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar \< > > > > [govi...@gmail.com](mailto:govi...@gmail.com)\> wrote: > > > > > > > > > I currently run my Elasticsearch cluster on 'Y' number of AWS EBS > > > > > backed reserved (reliable) instances. I'm now exploring the possibility of > > > > > introducing '3Y' AWS spot instances (unreliable but very cheap) with the > > > > > same specs into the system, turning them on and off depending on the load > > > > > on the cluster. > > > > > > > > > > So: > > > > > 'Y' reserved (reliable) machines > > > > > '3Y' spot (unreliable) machines > > > > > > > > > > I want at least one copy of each shard to be available on the 'Y' > > > > > reserved (reliable) machines, therefore I've placed all 'Y' reserved > > > > > instances in "rack1" and '3Y' spot instances in "rack2" with forced > > > > > awareness ([Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/ **** guide/reference/modules/)\*\* > > > > > cluste\*\*r.html[http://www.elasticsearch.org/guide/reference/modules/cluster.html](http://www.elasticsearch.org/guide/reference/modules/cluster.html)) > > > > > and one replica for each shard. > > > > > > > > > > Thus, the Y reserved instances will now have ~3 times as many shards > > > > > as the 3Y spot instances and "rack1" will cumulatively have lesser memory > > > > > and CPU. As a result, will searches that are directed to the 'Y' reserved > > > > > instances be slower than searches that are directed to the '3Y' spot > > > > > instances? > > > > > > > > Yes, searches that will hit the busy nodes should be slower. But you > > > > can fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and > > > > 4 as equal numbers within your poll of 3Y spot instances. Then make 3 > > > > replicas instead of 1 and you should have everything balanced. Plus, your > > > > cluster should handle more concurrent searches. > > > > > > > > > Or will a search directed at "rack1" (which, within itself, contains 1 > > > > > copy of each shard) be scattered/gathered across all nodes, including those > > > > > in "rack2"? > > > > > > > > AFAIK, ES simply round robins between the needed shards (primaries and > > > > replicas don't matter here, unless you specify that in your search > > > > preferences[0]). So it won't be aware that some shard will return results > > > > faster than others. > > > > > > > > [0] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/ **g** uide/reference/api/search/**pref)\* > > > > \*erence.html[http://www.elasticsearch.org/guide/reference/api/search/preference.html](http://www.elasticsearch.org/guide/reference/api/search/preference.html) > > > > > > > > ## Best regards, Radu > > > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > > -- > > > > -- > > > > -- -- ---
**Author:** ![govind201](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@govind201](https://discuss.elastic.co/u/govind201)\ **Post date:** [January 8, 2013, 4:54am UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/8 "2013-01-08T04:54:35Z")
Hi Radu, Here's a screenshot of my cluster - is this what you were looking for? "server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting down the nodes manually and seeing if this would help - this caused a reshuffle). [https://lh5.googleusercontent.com/-H5ygTL-4PfQ/UOulYhgnMhI/AAAAAAAAAhc/wo8WrC3nFKI/s1600/Screen+Shot+2013-01-07+at+8.48.59+PM.png](https://lh5.googleusercontent.com/-H5ygTL-4PfQ/UOulYhgnMhI/AAAAAAAAAhc/wo8WrC3nFKI/s1600/Screen+Shot+2013-01-07+at+8.48.59+PM.png) On Monday, January 7, 2013 12:09:13 AM UTC-8, Radu Gheorghe wrote: > Hello Govind, > > That's interesting, they should have been evened out... > > Can you share screenshot(s) from elasticsearch-head? Also, what version of > ES are you using? > > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > On Mon, Jan 7, 2013 at 4:09 AM, Govind Chandrasekhar \<[govi...@gmail.com](mailto:govi...@gmail.com)\ > > > wrote: > > > Thanks Radu and Shay. > > > > I did a test setup as follows to evaluate performance: > > > > - 91 shards. 3 replicas of each shard (including the primary). > > - 1 node in rack 1. > > - 1 node in rack 2. > > - 2 nodes in rack 3. > > > > Racks 1 and 2 seem fine. But the shards in rack 3 are imbalanced - one > > node has 30 shards and the other has 61 shards. I thought I'd get a 45-46 > > split. Am I missing something here? > > > > On Sunday, January 6, 2013 2:42:16 PM UTC-8, kimchy wrote: > > > > > Just an important note here, when using awareness allocation, and lets > > > say you "hit" a node on rack1 (i.e. send an HTTP search request that will > > > hit nodeX on rack1), it will prefer to execute the search "within the rack" > > > if possible, i.e., it will prefer copies of shards that exist on nodes > > > within rack1, compared to going "outside" of the rack. > > > > > > On Jan 5, 2013, at 5:08 PM, Radu Gheorghe [radu.g...@sematext.com](mailto:radu.g...@sematext.com) > > > wrote: > > > > > > Hi Govind, > > > > > > On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar [govi...@gmail.com](mailto:govi...@gmail.com)wrote: > > > > > > > Hi Radu, > > > > > > > > Thanks for that. Are there any good resources for better understanding > > > > how the query scattering and gathering process is conducted? > > > > > > I think this is a good place to look: > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/api/search/)\*\* > > > search-type.html[http://www.elasticsearch.org/guide/reference/api/search/search-type.html](http://www.elasticsearch.org/guide/reference/api/search/search-type.html) > > > > > > > If I were to create 3 spot racks + 1 reserved rack as you mentioned, > > > > will average query response time improve (aside from the benefit of support > > > > for more concurrent connections + larger cache) , i.e., will the query be > > > > scattered across all the racks, making use of CPU resources in all racks? > > > > > > One query is sent to one set of shards (whether they are primaries or > > > replicas it doesn't matter). So in the 4-rack scenario, if you run a single > > > query when all is quiet and we ignore the state of caches and other > > > possible variables, you should get the same X response time no matter which > > > shards your query hits. In your 2-rack scenario, I would assume queries > > > that hit shards with 3x nodes would be faster. So the average query time > > > should be better. > > > > > > However, if you have more than 2 concurrent queries on average, I would > > > expect the 4-rack scenario to get you better average times. And more > > > consistent times of course - because all nodes would have the same load. > > > Plus there's less chance of a bottleneck on your reserved instances. > > > > > > I'd also recommend to run a test and see the real-life results. And > > > observe the load, too. > > > > > > Also, are the caches of each of these nodes interlinked - if a filter is > > > > > > > cached in node A but the query is directed to node B, will the cached > > > > filter on node A be used or will the filter be rebuilt on node B? > > > > > > The filter will run on the selected shard and benefit from the cache of > > > the node where the shard is located. So node B won't benefit from the cache > > > on node A. > > > > > > You can warm up the caches if you use 0.20+ by using the Warmers API: > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/api/admin-)\*\* > > > indices-warmers.html[http://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html) > > > > > > ## Best regards, Radu > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > > > Govind > > > > > > > > On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote: > > > > > > > > > Hello Govind, > > > > > > > > > > On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar \< > > > > > [govi...@gmail.com](mailto:govi...@gmail.com)\> wrote: > > > > > > > > > > > I currently run my Elasticsearch cluster on 'Y' number of AWS EBS > > > > > > backed reserved (reliable) instances. I'm now exploring the possibility of > > > > > > introducing '3Y' AWS spot instances (unreliable but very cheap) with the > > > > > > same specs into the system, turning them on and off depending on the load > > > > > > on the cluster. > > > > > > > > > > > > So: > > > > > > 'Y' reserved (reliable) machines > > > > > > '3Y' spot (unreliable) machines > > > > > > > > > > > > I want at least one copy of each shard to be available on the 'Y' > > > > > > reserved (reliable) machines, therefore I've placed all 'Y' reserved > > > > > > instances in "rack1" and '3Y' spot instances in "rack2" with forced > > > > > > awareness ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ **** guide/reference/modules/)\* > > > > > > \*cluste\*\*r.html[http://www.elasticsearch.org/guide/reference/modules/cluster.html](http://www.elasticsearch.org/guide/reference/modules/cluster.html)) > > > > > > and one replica for each shard. > > > > > > > > > > > > Thus, the Y reserved instances will now have ~3 times as many shards > > > > > > as the 3Y spot instances and "rack1" will cumulatively have lesser memory > > > > > > and CPU. As a result, will searches that are directed to the 'Y' reserved > > > > > > instances be slower than searches that are directed to the '3Y' spot > > > > > > instances? > > > > > > > > > > Yes, searches that will hit the busy nodes should be slower. But you > > > > > can fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and > > > > > 4 as equal numbers within your poll of 3Y spot instances. Then make 3 > > > > > replicas instead of 1 and you should have everything balanced. Plus, your > > > > > cluster should handle more concurrent searches. > > > > > > > > > > > Or will a search directed at "rack1" (which, within itself, contains > > > > > > 1 copy of each shard) be scattered/gathered across all nodes, including > > > > > > those in "rack2"? > > > > > > > > > > AFAIK, ES simply round robins between the needed shards (primaries and > > > > > replicas don't matter here, unless you specify that in your search > > > > > preferences[0]). So it won't be aware that some shard will return results > > > > > faster than others. > > > > > > > > > > [0] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ **g** uide/reference/api/search/**pref) > > > > > \*\*erence.html[http://www.elasticsearch.org/guide/reference/api/search/preference.html](http://www.elasticsearch.org/guide/reference/api/search/preference.html) > > > > > > > > > > ## Best regards, Radu > > > > > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > > > > -- > > > > > > -- > > > > > > -- -- ---
**Author:** ![govind201](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@govind201](https://discuss.elastic.co/u/govind201)\ **Post date:** [January 8, 2013, 5:04am UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/9 "2013-01-08T05:04:03Z")
Oh, and I failed to mention - I'm using 0.19.8 On Monday, January 7, 2013 8:54:35 PM UTC-8, Govind Chandrasekhar wrote: > Hi Radu, > > Here's a screenshot of my cluster - is this what you were looking for? > "server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now > have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting > down the nodes manually and seeing if this would help - this caused a > reshuffle). > > [https://lh5.googleusercontent.com/-H5ygTL-4PfQ/UOulYhgnMhI/AAAAAAAAAhc/wo8WrC3nFKI/s1600/Screen+Shot+2013-01-07+at+8.48.59+PM.png](https://lh5.googleusercontent.com/-H5ygTL-4PfQ/UOulYhgnMhI/AAAAAAAAAhc/wo8WrC3nFKI/s1600/Screen+Shot+2013-01-07+at+8.48.59+PM.png) > > On Monday, January 7, 2013 12:09:13 AM UTC-8, Radu Gheorghe wrote: > > > Hello Govind, > > > > That's interesting, they should have been evened out... > > > > Can you share screenshot(s) from elasticsearch-head? Also, what version > > of ES are you using? > > > > ## Best regards, Radu > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > On Mon, Jan 7, 2013 at 4:09 AM, Govind Chandrasekhar [govi...@gmail.com](mailto:govi...@gmail.com)wrote: > > > > > Thanks Radu and Shay. > > > > > > I did a test setup as follows to evaluate performance: > > > > > > - 91 shards. 3 replicas of each shard (including the primary). > > > - 1 node in rack 1. > > > - 1 node in rack 2. > > > - 2 nodes in rack 3. > > > > > > Racks 1 and 2 seem fine. But the shards in rack 3 are imbalanced - one > > > node has 30 shards and the other has 61 shards. I thought I'd get a 45-46 > > > split. Am I missing something here? > > > > > > On Sunday, January 6, 2013 2:42:16 PM UTC-8, kimchy wrote: > > > > > > > Just an important note here, when using awareness allocation, and lets > > > > say you "hit" a node on rack1 (i.e. send an HTTP search request that will > > > > hit nodeX on rack1), it will prefer to execute the search "within the rack" > > > > if possible, i.e., it will prefer copies of shards that exist on nodes > > > > within rack1, compared to going "outside" of the rack. > > > > > > > > On Jan 5, 2013, at 5:08 PM, Radu Gheorghe [radu.g...@sematext.com](mailto:radu.g...@sematext.com) > > > > wrote: > > > > > > > > Hi Govind, > > > > > > > > On Sat, Jan 5, 2013 at 6:10 AM, Govind Chandrasekhar \<[govi...@gmail.com](mailto:govi...@gmail.com) > > > > > > > > > wrote: > > > > > > > > > Hi Radu, > > > > > > > > > > Thanks for that. Are there any good resources for better understanding > > > > > how the query scattering and gathering process is conducted? > > > > > > > > I think this is a good place to look: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/api/search/)\*\* > > > > search-type.html[http://www.elasticsearch.org/guide/reference/api/search/search-type.html](http://www.elasticsearch.org/guide/reference/api/search/search-type.html) > > > > > > > > > If I were to create 3 spot racks + 1 reserved rack as you mentioned, > > > > > will average query response time improve (aside from the benefit of support > > > > > for more concurrent connections + larger cache) , i.e., will the query be > > > > > scattered across all the racks, making use of CPU resources in all racks? > > > > > > > > One query is sent to one set of shards (whether they are primaries or > > > > replicas it doesn't matter). So in the 4-rack scenario, if you run a single > > > > query when all is quiet and we ignore the state of caches and other > > > > possible variables, you should get the same X response time no matter which > > > > shards your query hits. In your 2-rack scenario, I would assume queries > > > > that hit shards with 3x nodes would be faster. So the average query time > > > > should be better. > > > > > > > > However, if you have more than 2 concurrent queries on average, I would > > > > expect the 4-rack scenario to get you better average times. And more > > > > consistent times of course - because all nodes would have the same load. > > > > Plus there's less chance of a bottleneck on your reserved instances. > > > > > > > > I'd also recommend to run a test and see the real-life results. And > > > > observe the load, too. > > > > > > > > Also, are the caches of each of these nodes interlinked - if a filter > > > > > > > > > is cached in node A but the query is directed to node B, will the cached > > > > > filter on node A be used or will the filter be rebuilt on node B? > > > > > > > > The filter will run on the selected shard and benefit from the cache of > > > > the node where the shard is located. So node B won't benefit from the cache > > > > on node A. > > > > > > > > You can warm up the caches if you use 0.20+ by using the Warmers API: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/**guide/reference/api/admin-)\*\* > > > > indices-warmers.html[http://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-warmers.html) > > > > > > > > ## Best regards, Radu > > > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > > > > > Govind > > > > > > > > > > On Thursday, January 3, 2013 2:59:32 AM UTC-8, Radu Gheorghe wrote: > > > > > > > > > > > Hello Govind, > > > > > > > > > > > > On Mon, Dec 31, 2012 at 6:27 PM, Govind Chandrasekhar \< > > > > > > [govi...@gmail.com](mailto:govi...@gmail.com)\> wrote: > > > > > > > > > > > > > I currently run my Elasticsearch cluster on 'Y' number of AWS EBS > > > > > > > backed reserved (reliable) instances. I'm now exploring the possibility of > > > > > > > introducing '3Y' AWS spot instances (unreliable but very cheap) with the > > > > > > > same specs into the system, turning them on and off depending on the load > > > > > > > on the cluster. > > > > > > > > > > > > > > So: > > > > > > > 'Y' reserved (reliable) machines > > > > > > > '3Y' spot (unreliable) machines > > > > > > > > > > > > > > I want at least one copy of each shard to be available on the 'Y' > > > > > > > reserved (reliable) machines, therefore I've placed all 'Y' reserved > > > > > > > instances in "rack1" and '3Y' spot instances in "rack2" with forced > > > > > > > awareness ([Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ **** guide/reference/modules/) > > > > > > > **cluste** r.html[http://www.elasticsearch.org/guide/reference/modules/cluster.html](http://www.elasticsearch.org/guide/reference/modules/cluster.html)) > > > > > > > and one replica for each shard. > > > > > > > > > > > > > > Thus, the Y reserved instances will now have ~3 times as many shards > > > > > > > as the 3Y spot instances and "rack1" will cumulatively have lesser memory > > > > > > > and CPU. As a result, will searches that are directed to the 'Y' reserved > > > > > > > instances be slower than searches that are directed to the '3Y' spot > > > > > > > instances? > > > > > > > > > > > > Yes, searches that will hit the busy nodes should be slower. But you > > > > > > can fix that by having 4 racks: 1 with the Y reserved instances and 2,3 and > > > > > > 4 as equal numbers within your poll of 3Y spot instances. Then make 3 > > > > > > replicas instead of 1 and you should have everything balanced. Plus, your > > > > > > cluster should handle more concurrent searches. > > > > > > > > > > > > > Or will a search directed at "rack1" (which, within itself, contains > > > > > > > 1 copy of each shard) be scattered/gathered across all nodes, including > > > > > > > those in "rack2"? > > > > > > > > > > > > AFAIK, ES simply round robins between the needed shards (primaries > > > > > > and replicas don't matter here, unless you specify that in your search > > > > > > preferences[0]). So it won't be aware that some shard will return results > > > > > > faster than others. > > > > > > > > > > > > [0] [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/ **g** uide/reference/api/search/)\*\* > > > > > > pref\*\*erence.html[http://www.elasticsearch.org/guide/reference/api/search/preference.html](http://www.elasticsearch.org/guide/reference/api/search/preference.html) > > > > > > > > > > > > ## Best regards, Radu > > > > > > > > > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > > > > > > > > > > -- > > > > > > > > -- > > > > > > > > -- -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 9, 2013, 9:13am UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/10 "2013-01-09T09:13:03Z")
Hi Govind, On Tue, Jan 8, 2013 at 6:54 AM, Govind Chandrasekhar [govind201@gmail.com](mailto:govind201@gmail.com)wrote: > Hi Radu, > > Here's a screenshot of my cluster - is this what you were looking for? > "server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now > have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting > down the nodes manually and seeing if this would help - this caused a > reshuffle). Actually, I was interested in a screenshot from elasticsearch-head: [http://mobz.github.com/elasticsearch-head/](http://mobz.github.com/elasticsearch-head/) So I can see better which shard on which index is allocated to which node. I was also interested in how many indices you have and why you have so many total shards. While things are better now, you can check the same elasticsearch-head plugin to see if shards are still moving around. You can also get this information via the Cluster Health API: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... If shards are inbalanced and nothing is moving, you can check out the logs for clues. Ultimately, you can try to move shards manually using the Cluster Reroute API: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![govind201](https://avatars.discourse-cdn.com/v4/letter/g/8dc957/32.png) [@govind201](https://discuss.elastic.co/u/govind201)\ **Post date:** [January 9, 2013, 5:18pm UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/11 "2013-01-09T17:18:30Z")
Hi Radu, On Wednesday, January 9, 2013 1:13:03 AM UTC-8, Radu Gheorghe wrote: > Hi Govind, > > On Tue, Jan 8, 2013 at 6:54 AM, Govind Chandrasekhar \<[govi...@gmail.com](mailto:govi...@gmail.com)\ > > > wrote: > > > Hi Radu, > > > > Here's a screenshot of my cluster - is this what you were looking for? > > "server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now > > have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting > > down the nodes manually and seeing if this would help - this caused a > > reshuffle). > > Actually, I was interested in a screenshot from elasticsearch-head: > [http://mobz.github.com/elasticsearch-head/](http://mobz.github.com/elasticsearch-head/) > > So I can see better which shard on which index is allocated to which node. > I was also interested in how many indices you have and why you have so many > total shards. The elasticsearch-head screenshot's far too spread out because of the number of nodes unfortunately. I have 23 main indices - one for each product category -, with 3 shards each. The remaining indices - 22 shards - are either negligibly small (will the presence of empty shards cause an issue?). When a user makes a query, it is automatically directed to any one index (3 shards), though I am eventually hoping to support queries that span all 69 shards. > While things are better now, you can check the same elasticsearch-head > plugin to see if shards are still moving around. You can also get this > information via the Cluster Health API: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-cluster-health.html) > > If shards are inbalanced and nothing is moving, you can check out the logs > for clues. > > Ultimately, you can try to move shards manually using the Cluster Reroute > API: > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-cluster-reroute.html) > > Yup, this is what I"m resorting to! > ## Best regards, Radu > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![govind201\_2](https://avatars.discourse-cdn.com/v4/letter/g/a9adbd/32.png) [@govind201\_2](https://discuss.elastic.co/u/govind201_2)\ **Post date:** [January 16, 2013, 10:46pm UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/12 "2013-01-16T22:46:15Z")
_An update_: I've tried many combinations of racks with awareness, and the shards are usually unbalanced within machines of the same rack. I suspect that this a bug? Everything else works fine (I'm using the the cluster reroute API for the moment). On Wednesday, January 9, 2013 9:18:30 AM UTC-8, Govind Chandrasekhar wrote: > Hi Radu, > > On Wednesday, January 9, 2013 1:13:03 AM UTC-8, Radu Gheorghe wrote: > > > Hi Govind, > > > > On Tue, Jan 8, 2013 at 6:54 AM, Govind Chandrasekhar [govi...@gmail.com](mailto:govi...@gmail.com)wrote: > > > > > Hi Radu, > > > > > > Here's a screenshot of my cluster - is this what you were looking for? > > > "server-3" and "server-4" have 91 shards each. But spot-5 and spot-6 now > > > have 50 and 41 respectively (it was earlier 60-31, but I'd tried shutting > > > down the nodes manually and seeing if this would help - this caused a > > > reshuffle). > > > > Actually, I was interested in a screenshot from elasticsearch-head: > > [http://mobz.github.com/elasticsearch-head/](http://mobz.github.com/elasticsearch-head/) > > > > So I can see better which shard on which index is allocated to which > > node. I was also interested in how many indices you have and why you have > > so many total shards. > > The elasticsearch-head screenshot's far too spread out because of the > number of nodes unfortunately. I have 23 main indices - one for each > product category -, with 3 shards each. The remaining indices - 22 shards - > are either negligibly small (will the presence of empty shards cause an > issue?). When a user makes a query, it is automatically directed to any one > index (3 shards), though I am eventually hoping to support queries that > span all 69 shards. > > > While things are better now, you can check the same elasticsearch-head > > plugin to see if shards are still moving around. You can also get this > > information via the Cluster Health API: > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-cluster-health.html) > > > > If shards are inbalanced and nothing is moving, you can check out the > > logs for clues. > > > > Ultimately, you can try to move shards manually using the Cluster Reroute > > API: > > > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-cluster-reroute.html) > > > > Yup, this is what I"m resorting to! > > > ## Best regards, Radu > > > > [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![VincentBiret](https://avatars.discourse-cdn.com/v4/letter/v/54ee81/32.png) [@VincentBiret](https://discuss.elastic.co/u/VincentBiret)\ **Post date:** [May 26, 2016, 7:27pm UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/13 "2016-05-26T19:27:20Z")
> [@radu\_gheorghe](#): > > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/api/search/preference.html) is dead [search request preference](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-preference.html) I have been looking for this link for several hours, thanks ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:48pm UTC](https://discuss.elastic.co/t/aws-spot-reserved-instances/10208/14 "2017-07-05T22:48:46Z")