# Question: shard awareness allocation and shard allocation filtering **URL:** \ **Category:** Elasticsearch\ **Created:** [October 16, 2014, 9:42am UTC](https://discuss.elastic.co/t/question-shard-awareness-allocation-and-shard-allocation-filtering/20280 "2014-10-16T09:42:13Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Gregoire\_Seux1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregoire_seux1/32/1169_2.png) [@Gregoire\_Seux1](https://discuss.elastic.co/u/Gregoire_Seux1)\ **Post date:** [October 16, 2014, 9:42am UTC](https://discuss.elastic.co/t/question-shard-awareness-allocation-and-shard-allocation-filtering/20280/1 "2014-10-16T09:42:13Z")
Hello, My question is related to cooperation between shard awareness allocation and shard allocation filtering. We have two kinds of nodes: those with ssds (used for indexing and search recent data), those with large spinning disks (used for archiving old indices). I'd like to setup a mechanism to move old indices from ssds to spinning disks. The first solution uses reroute command in cluster api. However it feels unnatural since you have to do it shard by shard and decide the target node. What I want to achieve is the following: 1. stick recent indices (the current one being written) to ssds. They have 2 copies. 2. at some point (disk on ssds is above 65%), one copy is moved to larger boxes (1 copy is still on ssd to help search, 1 copy on large box) 3. when disk is scarce on ssd boxes (90%), we simply drop the copy present on ssd. Since we don't care that much of old data having only one copy is not an issue. I have tried to implement this with shard awareness allocation and allocation filtering but it does not seem to work as expected. Nodes have _flavor_ attribute depending on their hardware (_ssd_ or _iodisk_ ). Cluster is using shard awareness based on _flavor_ attribute (_cluster.routing.allocation.awareness.attributes: flavor_). 1. My index template has _routing.allocation.require: ssd_ to impose two have all copies on ssds first. 2. At some point, I drop the requirement (effectively \*routing.allocation.require: \*\*). I expect flavor awareness to move one copy to large (iodisk) boxes. 3. At a later point, I'll set _number\_of\_replicas_ to 0 and change _routing.allocation.require_ to _iodisk_ to drop the shard copy on ssds Sadly allocation filtering and shard awareness do not seem to cooperate well : when an new index is created, one copy goes to ssds and the other is not allocated anywhere (index stays in yellow state). Using _curl -XPUT localhost:9200/\_cluster/settings -d '{"transient":{"logger.cluster.routing.allocation":"trace"}}_, I have observed what happen when a new index is created. [2014-10-16 06:53:19,462][TRACE][cluster.routing.allocation.decider] > [bungeearchive01-par.storage.criteo.preprod] Can not allocate > [[2014-10-16.01][3], node[null], [R], s[UNASSIGNED]] on node > [qK34VLdhTferCQs2oNJOyg] due to [SameShardAllocationDecider] > [2014-10-16 06:53:19,463][TRACE][cluster.routing.allocation.decider] > [bungeearchive01-par.storage.criteo.preprod] Can not allocate > [[2014-10-16.01][3], node[null], [R], s[UNASSIGNED]] on node > [gE7OTgevSUuoj44RozxK0Q] due to [AwarenessAllocationDecider] > [2014-10-16 06:53:19,463][TRACE][cluster.routing.allocation.decider] > [bungeearchive01-par.storage.criteo.preprod] Can not allocate > [[2014-10-16.01][3], node[null], [R], s[UNASSIGNED]] on node > [Y2k9qXfsTx6X2iQTxg9RBQ] due to [AwarenessAllocationDecider] > [2014-10-16 06:53:19,463][TRACE][cluster.routing.allocation.decider] > [bungeearchive01-par.storage.criteo.preprod] Can not allocate > [[2014-10-16.01][3], node[null], [R], s[UNASSIGNED]] on node > [FwWc2XPPRWuje2KH6AlDEQ] due to [FilterAllocationDecider] > [2014-10-16 06:53:19,492][TRACE][cluster.routing.allocation.allocator] > [bungeearchive01-par.storage.criteo.preprod] No Node found to assign shard > [[2014-10-16.01][3], node[null], [R], s[UNASSIGNED]] This transcript shows that - shard 3 primary replica is on node qK34VLdhTferCQs2oNJOyg (flavor:ssd) which prevent its copy to placed there - it cannot be placed on gE7OTgevSUuoj44RozxK0Q (ssd as well) because it tries to maximizes dispersion accross flavors - it cannot be placed on Y2k9qXfsTx6X2iQTxg9RBQ for the same reason - it cannot be placed on FwWc2XPPRWuje2KH6AlDEQ (flavor: iodisk) because of the filter Questions: - am I doing it wrong? - should I stick with a set of reroute command? - are awareness and filtering supposed to cooperate? Any help will be appreciated -- Grégoire Seux -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/61665ede-f772-43c5-b159-53bf9353bbdf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/61665ede-f772-43c5-b159-53bf9353bbdf%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Gregoire\_Seux1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregoire_seux1/32/1169_2.png) [@Gregoire\_Seux1](https://discuss.elastic.co/u/Gregoire_Seux1)\ **Post date:** [October 18, 2014, 6:36pm UTC](https://discuss.elastic.co/t/question-shard-awareness-allocation-and-shard-allocation-filtering/20280/2 "2014-10-18T18:36:52Z")
On Thu, Oct 16, 2014 at 11:42 AM, Grégoire Seux [kamaradclimber@gmail.com](mailto:kamaradclimber@gmail.com) wrote: > - are awareness and filtering supposed to cooperate? A quick look at the code confirm that allocation deciders are fully orthogonal. Should I open a github issue to discuss adding support for cooperating deciders ? -- Grégoire Seux -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAE3ghBu%2BiCENFsMBWrE0UwiKOikcKyRox4eGBChAFVK78%3DY-OA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAE3ghBu%2BiCENFsMBWrE0UwiKOikcKyRox4eGBChAFVK78%3DY-OA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Boaz\_Leskes](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boaz_leskes/32/723_2.png) [@Boaz\_Leskes](https://discuss.elastic.co/u/Boaz_Leskes)\ **Post date:** [October 20, 2014, 12:01pm UTC](https://discuss.elastic.co/t/question-shard-awareness-allocation-and-shard-allocation-filtering/20280/3 "2014-10-20T12:01:31Z")
Hi Grégoire A couple of comments: > 1. at some point (disk on ssds is above 65%), one copy is moved to larger > boxes (1 copy is still on ssd to help search, 1 copy on large box) Allocation awareness causes elasticsearch to spread the shards copies across the different values of the attribute. However, it also changes the search behavior in the sense that it tries to execute searches on nodes that have the same attributes as the one that initially got the search. In your case it means that if an ssd node got the search, it will run on SSD otherwise it will on iodisk. I'm not sure this is what you want. > 1. At some point, I drop the requirement (effectively \*routing.allocation.require: > \*\*). I expect flavor awareness to move one copy to large (iodisk) boxes. ES tries the balance shards from the cluster perspective. It gives some weight to spreading up the shards of an index but this just one parameter.In your cases I suspect you have way more shards on the iodisk nodes than on the ssds, which means that balancing will try to move shards from iodisks to ssds if it can but not the other way around (as you expect). > are awareness and filtering supposed to cooperate? I think they should but I'm not sure it will achieve what you want to do - see comment above. That said, I can confirm that shard allocation awareness and filtering on the same attribute may be in each other way. I would suggest you open an issue on github indicating that when shard allocation awareness is causing unassigned shards if one of the attribute values is blocked by an allocation filter (doesn't matter which filter is being used). You would expect it to behave the same as if the nodes were down (in which case the shards will be assigned). Try to give a concise reproduction using two different attributes for filtering and awareness. Cheers, Boaz On Saturday, October 18, 2014 8:37:29 PM UTC+2, Grégoire Seux wrote: > On Thu, Oct 16, 2014 at 11:42 AM, Grégoire Seux > [kamaradclimber@gmail.com](mailto:kamaradclimber@gmail.com) wrote: > > > - are awareness and filtering supposed to cooperate? > > A quick look at the code confirm that allocation deciders are fully > orthogonal. > Should I open a github issue to discuss adding support for cooperating > deciders ? > > -- > Grégoire Seux -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b6cc56d4-f2aa-403c-a46e-54c34b3a41a9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b6cc56d4-f2aa-403c-a46e-54c34b3a41a9%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Gregoire\_Seux1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gregoire_seux1/32/1169_2.png) [@Gregoire\_Seux1](https://discuss.elastic.co/u/Gregoire_Seux1)\ **Post date:** [October 21, 2014, 1:29pm UTC](https://discuss.elastic.co/t/question-shard-awareness-allocation-and-shard-allocation-filtering/20280/4 "2014-10-21T13:29:37Z")
Hi Boaz, Thanks for your answer. > In your case it means that if an ssd node got the search, it will run > on SSD otherwise it will on iodisk. I'm not sure this is what you > want. This is the effect I am looking for. During the period where I have a copy on ssds and on iodisks, I prefer to query the fastest ones. > ES tries the balance shards from the cluster perspective. It gives some > weight to spreading up the shards of an index but this just one > parameter.In your cases I suspect you have way more shards on the iodisk > nodes than on the ssds, which means that balancing will try to move shards > from iodisks to ssds if it can but not the other way around (as you expect). You're right, I'd like to have way more shards on iodisk indeed. Reading [https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/AllocationDecidersModule.java#L66](https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/cluster/routing/allocation/decider/AllocationDecidersModule.java#L66), I don't see any filter that will prevent allocation from ssd to iodisk. I understand that ES will see the situation as unbalanced and will try to rebalance it but I expect allocation filter to prevent those backward reallocation. > I would suggest you open an issue on github indicating that when shard > allocation awareness is causing unassigned shards if one of the > attribute values is blocked by an allocation filter Done: [rack awareness allocation and allocation filtering lead to unassigned shards · Issue #8178 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/8178) -- Grégoire -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/20141021132937.GB1762%40criteo-scalasto.criteo.prod](https://groups.google.com/d/msgid/elasticsearch/20141021132937.GB1762%40criteo-scalasto.criteo.prod). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:54am UTC](https://discuss.elastic.co/t/question-shard-awareness-allocation-and-shard-allocation-filtering/20280/5 "2017-07-06T00:54:42Z")