# Shard allocation logic not taking disk size into account - why? **URL:** \ **Category:** Elasticsearch\ **Created:** [July 10, 2013, 1:43pm UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743 "2013-07-10T13:43:20Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [July 10, 2013, 1:43pm UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/1 "2013-07-10T13:43:20Z")
Hi, I'm aware of [https://github.com/sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) But is there a reason why similar functionality isn't implemented in ElasticSearch itself? In particular, ES will try and allocate shards on machines which are low on disk. We are seeing this all the time and I have to relocate shards manually to prevent that. Why isn't there code in place to prevent that? Itamar. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vadim\_Kisselmann](https://avatars.discourse-cdn.com/v4/letter/v/2bfe46/32.png) [@Vadim\_Kisselmann](https://discuss.elastic.co/u/Vadim_Kisselmann)\ **Post date:** [July 10, 2013, 2:30pm UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/2 "2013-07-10T14:30:30Z")
Hi, because ES relocate shards when your disk is full automatically. We monitored an "excellent" behavior of ES last time, than all our AWS nodes are running out of disk space. The full index was held in network(rebalancing) till a new node was joined the cluster. Cheers, Vadim Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko: > Hi, > > I'm aware of [GitHub - sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > But is there a reason why similar functionality isn't implemented in > Elasticsearch itself? > > In particular, ES will try and allocate shards on machines which are low > on disk. We are seeing this all the time and I have to relocate shards > manually to prevent that. Why isn't there code in place to prevent that? > > Itamar. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [July 11, 2013, 2:23am UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/3 "2013-07-11T02:23:10Z")
Hi Vadim, I can't tell whether "excellent" is meant as emphasis or sarcasm? What do you mean by "full index was help in network (rebalancing)"? It almost sounds like you are saying shards were being bounced around the cluster, but maybe you meant something else? ## Otis ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote: > Hi, > because ES relocate shards when your disk is full automatically. We > monitored an "excellent" behavior of ES last time, than all our AWS nodes > are running out of disk space. The full index was held in > network(rebalancing) till a new node was joined the cluster. > Cheers, > Vadim > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko: > > > Hi, > > > > I'm aware of [GitHub - sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > But is there a reason why similar functionality isn't implemented in > > Elasticsearch itself? > > > > In particular, ES will try and allocate shards on machines which are low > > on disk. We are seeing this all the time and I have to relocate shards > > manually to prevent that. Why isn't there code in place to prevent that? > > > > Itamar. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Vadim\_Kisselmann](https://avatars.discourse-cdn.com/v4/letter/v/2bfe46/32.png) [@Vadim\_Kisselmann](https://discuss.elastic.co/u/Vadim_Kisselmann)\ **Post date:** [July 11, 2013, 6:55am UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/4 "2013-07-11T06:55:12Z")
Hi Otis, i meant both with "excellent". Excellent for us, because our data was still there. Not excellent for our AWS bill, we had an traffic over 100TB in a couple of days. The shards were really bounced around the cluster because every node was running out of disk space. So ES tried to relocate the shards all the time. Cheers, Vadim Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic: > Hi Vadim, > > I can't tell whether "excellent" is meant as emphasis or sarcasm? > > What do you mean by "full index was help in network (rebalancing)"? It > almost sounds like you are saying shards were being bounced around the > cluster, but maybe you meant something else? > > ## Otis > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > > On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote: > > > Hi, > > because ES relocate shards when your disk is full automatically. We > > monitored an "excellent" behavior of ES last time, than all our AWS nodes > > are running out of disk space. The full index was held in > > network(rebalancing) till a new node was joined the cluster. > > Cheers, > > Vadim > > > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko: > > > > > Hi, > > > > > > I'm aware of [GitHub - sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > > > But is there a reason why similar functionality isn't implemented in > > > Elasticsearch itself? > > > > > > In particular, ES will try and allocate shards on machines which are low > > > on disk. We are seeing this all the time and I have to relocate shards > > > manually to prevent that. Why isn't there code in place to prevent that? > > > > > > Itamar. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [July 21, 2013, 11:11am UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/5 "2013-07-21T11:11:15Z")
Unfortunately we are seeing very undesired behavior - I do hope Elasticsearch developers will act quickly on this as this seems like a huge show stopper Yes, Elasticsearch will not lose data (at least no data loss that we were able to observe), but the current shard allocation behavior has some very undesirable effects We have couple of dozen of nodes, and whenever we add more nodes the cluster tries to rebalance. Whenever this occurs, it is very easy to notice a very flawed balancing function is used, and it takes a couple of manual shard allocation commands to stabilize the cluster again. We have some 400+ indexes with one shard each and some with multiple replicas, some very big (10gb at least in size), some medium (around 5gb) and many very small ones. Apparently the sharding function tries to balance the cluster based of number of indexes on a server, without taking into account its size on disk or number of documents in it. I has come to a point where we get to 0kb free on some of the nodes during normal operation and ES unable to flush to disk. This is pretty much unacceptable, considering we have nodes with 150gb+ free in the same cluster. At the very least nodes should refuse to accept new shards if they can't fit them in the FS taking into account other allocated (but not yet assigned) shards On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann [v.kisselmann@gmail.com](mailto:v.kisselmann@gmail.com)wrote: > Hi Otis, > i meant both with "excellent". Excellent for us, because our data was > still there. Not excellent for our AWS bill, we had an traffic over 100TB > in a couple of days. > The shards were really bounced around the cluster because every node was > running out of disk space. So ES tried to relocate the shards all the time. > Cheers, > Vadim > > Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic: > > > Hi Vadim, > > > > I can't tell whether "excellent" is meant as emphasis or sarcasm? > > > > What do you mean by "full index was help in network (rebalancing)"? It > > almost sounds like you are saying shards were being bounced around the > > cluster, but maybe you meant something else? > > > > ## Otis > > > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/)\*\* > > index.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote: > > > > > Hi, > > > because ES relocate shards when your disk is full automatically. We > > > monitored an "excellent" behavior of ES last time, than all our AWS nodes > > > are running out of disk space. The full index was held in > > > network(rebalancing) till a new node was joined the cluster. > > > Cheers, > > > Vadim > > > > > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko: > > > > > > > Hi, > > > > > > > > I'm aware of [https://github.com/sonian/\*\*elasticsearch-equilibrium](https://github.com/sonian/**elasticsearch-equilibrium)[https://github.com/sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > > > > > But is there a reason why similar functionality isn't implemented in > > > > Elasticsearch itself? > > > > > > > > In particular, ES will try and allocate shards on machines which are > > > > low on disk. We are seeing this all the time and I have to relocate shards > > > > manually to prevent that. Why isn't there code in place to prevent that? > > > > > > > > Itamar. > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [July 25, 2013, 10:49am UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/6 "2013-07-25T10:49:30Z")
Hi Itamar, Sounds scary... did you open an issue we could watch? ## Otis ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote: > Unfortunately we are seeing very undesired behavior - I do hope > Elasticsearch developers will act quickly on this as this seems like a huge > show stopper > > Yes, Elasticsearch will not lose data (at least no data loss that we were > able to observe), but the current shard allocation behavior has some very > undesirable effects > > We have couple of dozen of nodes, and whenever we add more nodes the > cluster tries to rebalance. Whenever this occurs, it is very easy to notice > a very flawed balancing function is used, and it takes a couple of manual > shard allocation commands to stabilize the cluster again. > > We have some 400+ indexes with one shard each and some with multiple > replicas, some very big (10gb at least in size), some medium (around 5gb) > and many very small ones. Apparently the sharding function tries to balance > the cluster based of number of indexes on a server, without taking into > account its size on disk or number of documents in it. > > I has come to a point where we get to 0kb free on some of the nodes during > normal operation and ES unable to flush to disk. This is pretty much > unacceptable, considering we have nodes with 150gb+ free in the same > cluster. > > At the very least nodes should refuse to accept new shards if they can't > fit them in the FS taking into account other allocated (but not yet > assigned) shards > > On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann \<[v.kiss...@gmail.com](mailto:v.kiss...@gmail.com)\ > > > wrote: > > > Hi Otis, > > i meant both with "excellent". Excellent for us, because our data was > > still there. Not excellent for our AWS bill, we had an traffic over 100TB > > in a couple of days. > > The shards were really bounced around the cluster because every node was > > running out of disk space. So ES tried to relocate the shards all the time. > > Cheers, > > Vadim > > > > Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic: > > > > > Hi Vadim, > > > > > > I can't tell whether "excellent" is meant as emphasis or sarcasm? > > > > > > What do you mean by "full index was help in network (rebalancing)"? It > > > almost sounds like you are saying shards were being bounced around the > > > cluster, but maybe you meant something else? > > > > > > ## Otis > > > > > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/)\*\* > > > index.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote: > > > > > > > Hi, > > > > because ES relocate shards when your disk is full automatically. We > > > > monitored an "excellent" behavior of ES last time, than all our AWS nodes > > > > are running out of disk space. The full index was held in > > > > network(rebalancing) till a new node was joined the cluster. > > > > Cheers, > > > > Vadim > > > > > > > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko: > > > > > > > > > Hi, > > > > > > > > > > I'm aware of [https://github.com/sonian/\*\*elasticsearch-equilibrium](https://github.com/sonian/**elasticsearch-equilibrium)[https://github.com/sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > > > > > > > But is there a reason why similar functionality isn't implemented in > > > > > Elasticsearch itself? > > > > > > > > > > In particular, ES will try and allocate shards on machines which are > > > > > low on disk. We are seeing this all the time and I have to relocate shards > > > > > manually to prevent that. Why isn't there code in place to prevent that? > > > > > > > > > > Itamar. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google Groups > > > > "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send an > > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [July 25, 2013, 11:23am UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/7 "2013-07-25T11:23:37Z")
No, not sure what issue that would be. Also haven't heard back on this thread or on other issues I've opened, so.... On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic \< [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > Hi Itamar, > > Sounds scary... did you open an issue we could watch? > > ## Otis > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > > On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote: > > > Unfortunately we are seeing very undesired behavior - I do hope > > Elasticsearch developers will act quickly on this as this seems like a huge > > show stopper > > > > Yes, Elasticsearch will not lose data (at least no data loss that we were > > able to observe), but the current shard allocation behavior has some very > > undesirable effects > > > > We have couple of dozen of nodes, and whenever we add more nodes the > > cluster tries to rebalance. Whenever this occurs, it is very easy to notice > > a very flawed balancing function is used, and it takes a couple of manual > > shard allocation commands to stabilize the cluster again. > > > > We have some 400+ indexes with one shard each and some with multiple > > replicas, some very big (10gb at least in size), some medium (around 5gb) > > and many very small ones. Apparently the sharding function tries to balance > > the cluster based of number of indexes on a server, without taking into > > account its size on disk or number of documents in it. > > > > I has come to a point where we get to 0kb free on some of the nodes > > during normal operation and ES unable to flush to disk. This is pretty much > > unacceptable, considering we have nodes with 150gb+ free in the same > > cluster. > > > > At the very least nodes should refuse to accept new shards if they can't > > fit them in the FS taking into account other allocated (but not yet > > assigned) shards > > > > On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann [v.kiss...@gmail.com](mailto:v.kiss...@gmail.com)wrote: > > > > > Hi Otis, > > > i meant both with "excellent". Excellent for us, because our data was > > > still there. Not excellent for our AWS bill, we had an traffic over 100TB > > > in a couple of days. > > > The shards were really bounced around the cluster because every node was > > > running out of disk space. So ES tried to relocate the shards all the time. > > > Cheers, > > > Vadim > > > > > > Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic: > > > > > > > Hi Vadim, > > > > > > > > I can't tell whether "excellent" is meant as emphasis or sarcasm? > > > > > > > > What do you mean by "full index was help in network (rebalancing)"? It > > > > almost sounds like you are saying shards were being bounced around the > > > > cluster, but maybe you meant something else? > > > > > > > > ## Otis > > > > > > > > ELASTICSEARCH Performance Monitoring - [http://sematext.com/spm/\*\*inde](http://sematext.com/spm/**inde)\*\* > > > > x.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > Search Analytics - [http://sematext.com/search-\*\*a\*\*nalytics/index.html](http://sematext.com/search- **a** nalytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > > > On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote: > > > > > > > > > Hi, > > > > > because ES relocate shards when your disk is full automatically. We > > > > > monitored an "excellent" behavior of ES last time, than all our AWS nodes > > > > > are running out of disk space. The full index was held in > > > > > network(rebalancing) till a new node was joined the cluster. > > > > > Cheers, > > > > > Vadim > > > > > > > > > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko: > > > > > > > > > > > Hi, > > > > > > > > > > > > I'm aware of [https://github.com/sonian/\*\*e\*\*lasticsearch-equilibrium](https://github.com/sonian/ **e** lasticsearch-equilibrium)[https://github.com/sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > > > > > > > > > But is there a reason why similar functionality isn't implemented in > > > > > > Elasticsearch itself? > > > > > > > > > > > > In particular, ES will try and allocate shards on machines which are > > > > > > low on disk. We are seeing this all the time and I have to relocate shards > > > > > > manually to prevent that. Why isn't there code in place to prevent that? > > > > > > > > > > > > Itamar. > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [July 26, 2013, 4:44pm UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/8 "2013-07-26T16:44:16Z")
Merging in Lucene can take up a considerable amount of disk space for intermediate segments depending on the merge settings. It would be hard to take disk size into consideration for allocation without thinking of the merge policy as well. I can see this issue being exacerbated with single-shard indices (which luckily I do not have). -- Ivan On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko [itamar@code972.com](mailto:itamar@code972.com)wrote: > No, not sure what issue that would be. Also haven't heard back on this > thread or on other issues I've opened, so.... > > On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic \< > [otis.gospodnetic@gmail.com](mailto:otis.gospodnetic@gmail.com)\> wrote: > > > Hi Itamar, > > > > Sounds scary... did you open an issue we could watch? > > > > ## Otis > > > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > > > > On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > Unfortunately we are seeing very undesired behavior - I do hope > > > Elasticsearch developers will act quickly on this as this seems like a huge > > > show stopper > > > > > > Yes, Elasticsearch will not lose data (at least no data loss that we > > > were able to observe), but the current shard allocation behavior has some > > > very undesirable effects > > > > > > We have couple of dozen of nodes, and whenever we add more nodes the > > > cluster tries to rebalance. Whenever this occurs, it is very easy to notice > > > a very flawed balancing function is used, and it takes a couple of manual > > > shard allocation commands to stabilize the cluster again. > > > > > > We have some 400+ indexes with one shard each and some with multiple > > > replicas, some very big (10gb at least in size), some medium (around 5gb) > > > and many very small ones. Apparently the sharding function tries to balance > > > the cluster based of number of indexes on a server, without taking into > > > account its size on disk or number of documents in it. > > > > > > I has come to a point where we get to 0kb free on some of the nodes > > > during normal operation and ES unable to flush to disk. This is pretty much > > > unacceptable, considering we have nodes with 150gb+ free in the same > > > cluster. > > > > > > At the very least nodes should refuse to accept new shards if they can't > > > fit them in the FS taking into account other allocated (but not yet > > > assigned) shards > > > > > > On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann [v.kiss...@gmail.com](mailto:v.kiss...@gmail.com)wrote: > > > > > > > Hi Otis, > > > > i meant both with "excellent". Excellent for us, because our data was > > > > still there. Not excellent for our AWS bill, we had an traffic over 100TB > > > > in a couple of days. > > > > The shards were really bounced around the cluster because every node > > > > was running out of disk space. So ES tried to relocate the shards all the > > > > time. > > > > Cheers, > > > > Vadim > > > > > > > > Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic: > > > > > > > > > Hi Vadim, > > > > > > > > > > I can't tell whether "excellent" is meant as emphasis or sarcasm? > > > > > > > > > > What do you mean by "full index was help in network (rebalancing)"? > > > > > It almost sounds like you are saying shards were being bounced around the > > > > > cluster, but maybe you meant something else? > > > > > > > > > > ## Otis > > > > > > > > > > ELASTICSEARCH Performance Monitoring - [http://sematext.com/spm/\*\*inde](http://sematext.com/spm/**inde)\* > > > > > \*x.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > > Search Analytics - [http://sematext.com/search-\*\*a\*\*nalytics/index.html](http://sematext.com/search- **a** nalytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > > > > > On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote: > > > > > > > > > > > Hi, > > > > > > because ES relocate shards when your disk is full automatically. We > > > > > > monitored an "excellent" behavior of ES last time, than all our AWS nodes > > > > > > are running out of disk space. The full index was held in > > > > > > network(rebalancing) till a new node was joined the cluster. > > > > > > Cheers, > > > > > > Vadim > > > > > > > > > > > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I'm aware of [https://github.com/sonian/\*\*e\*\*lasticsearch-equilibrium](https://github.com/sonian/ **e** lasticsearch-equilibrium)[https://github.com/sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > > > > > > > > > > > But is there a reason why similar functionality isn't implemented > > > > > > > in Elasticsearch itself? > > > > > > > > > > > > > > In particular, ES will try and allocate shards on machines which are > > > > > > > low on disk. We are seeing this all the time and I have to relocate shards > > > > > > > manually to prevent that. Why isn't there code in place to prevent that? > > > > > > > > > > > > > > Itamar. > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [July 26, 2013, 9:18pm UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/9 "2013-07-26T21:18:43Z")
I think you need to ensure that larger indexes have more shards, so the data can be more easily distributed. If you are running 0.90+ it should automatically ensure that shards for an index are evenly (for the most part distributed): > > > The current EvenShardCountAllocator balances shards in a global fashion across a…ll eligible nodes in the cluster. Under certain circumstances this algorithm can overload certain nodes with multiple shards for a single index while still gaining global balance. > The shard allocation algorithm should be more flexible to take into account more information like indexes or primaries and eventually things like the size of a shard or request patterns of an index. If that doesn't work for you, you can manually force max N shards per node with this setting: index.routing.allocation.total\_shards\_per\_node Best Regards, Paul On Friday, July 26, 2013 10:44:16 AM UTC-6, Ivan Brusic wrote: > Merging in Lucene can take up a considerable amount of disk space for > intermediate segments depending on the merge settings. It would be hard to > take disk size into consideration for allocation without thinking of the > merge policy as well. I can see this issue being exacerbated with > single-shard indices (which luckily I do not have). > > -- > Ivan > > On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko \<[ita...@code972.com](mailto:ita...@code972.com)\ > > > wrote: > > > No, not sure what issue that would be. Also haven't heard back on this > > thread or on other issues I've opened, so.... > > > > On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic \<[otis.gos...@gmail.com](mailto:otis.gos...@gmail.com)\ > > > > > wrote: > > > > > Hi Itamar, > > > > > > Sounds scary... did you open an issue we could watch? > > > > > > ## Otis > > > > > > ELASTICSEARCH Performance Monitoring - > > > [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) > > > Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) > > > > > > On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > > > Unfortunately we are seeing very undesired behavior - I do hope > > > > Elasticsearch developers will act quickly on this as this seems like a huge > > > > show stopper > > > > > > > > Yes, Elasticsearch will not lose data (at least no data loss that we > > > > were able to observe), but the current shard allocation behavior has some > > > > very undesirable effects > > > > > > > > We have couple of dozen of nodes, and whenever we add more nodes the > > > > cluster tries to rebalance. Whenever this occurs, it is very easy to notice > > > > a very flawed balancing function is used, and it takes a couple of manual > > > > shard allocation commands to stabilize the cluster again. > > > > > > > > We have some 400+ indexes with one shard each and some with multiple > > > > replicas, some very big (10gb at least in size), some medium (around 5gb) > > > > and many very small ones. Apparently the sharding function tries to balance > > > > the cluster based of number of indexes on a server, without taking into > > > > account its size on disk or number of documents in it. > > > > > > > > I has come to a point where we get to 0kb free on some of the nodes > > > > during normal operation and ES unable to flush to disk. This is pretty much > > > > unacceptable, considering we have nodes with 150gb+ free in the same > > > > cluster. > > > > > > > > At the very least nodes should refuse to accept new shards if they > > > > can't fit them in the FS taking into account other allocated (but not yet > > > > assigned) shards > > > > > > > > On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann [v.kiss...@gmail.com](mailto:v.kiss...@gmail.com)wrote: > > > > > > > > > Hi Otis, > > > > > i meant both with "excellent". Excellent for us, because our data was > > > > > still there. Not excellent for our AWS bill, we had an traffic over 100TB > > > > > in a couple of days. > > > > > The shards were really bounced around the cluster because every node > > > > > was running out of disk space. So ES tried to relocate the shards all the > > > > > time. > > > > > Cheers, > > > > > Vadim > > > > > > > > > > Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic: > > > > > > > > > > > Hi Vadim, > > > > > > > > > > > > I can't tell whether "excellent" is meant as emphasis or sarcasm? > > > > > > > > > > > > What do you mean by "full index was help in network (rebalancing)"? > > > > > > It almost sounds like you are saying shards were being bounced around the > > > > > > cluster, but maybe you meant something else? > > > > > > > > > > > > ## Otis > > > > > > > > > > > > ELASTICSEARCH Performance Monitoring - [http://sematext.com/spm/\*\*inde](http://sematext.com/spm/**inde) > > > > > > **x.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > > > Search Analytics - [http://sematext.com/search-\*\*a](http://sematext.com/search-**a)** > > > > > > nalytics/index.html [http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > > > > > > > On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote: > > > > > > > > > > > > > Hi, > > > > > > > because ES relocate shards when your disk is full automatically. We > > > > > > > monitored an "excellent" behavior of ES last time, than all our AWS nodes > > > > > > > are running out of disk space. The full index was held in > > > > > > > network(rebalancing) till a new node was joined the cluster. > > > > > > > Cheers, > > > > > > > Vadim > > > > > > > > > > > > > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I'm aware of [https://github.com/sonian/\*\*e](https://github.com/sonian/**e)\*\* > > > > > > > > lasticsearch-equilibrium[https://github.com/sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > > > > > > > > > > > > > But is there a reason why similar functionality isn't implemented > > > > > > > > in Elasticsearch itself? > > > > > > > > > > > > > > > > In particular, ES will try and allocate shards on machines which > > > > > > > > are low on disk. We are seeing this all the time and I have to relocate > > > > > > > > shards manually to prevent that. Why isn't there code in place to prevent > > > > > > > > that? > > > > > > > > > > > > > > > > Itamar. > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > . > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [August 4, 2013, 2:33pm UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/10 "2013-08-04T14:33:34Z")
Ivan, I hear you, but at the very least some free space calculation needs to be in effect. For example, when ES hits a lower limit (say, 500MB of free space) it should start relocation out or cancel relocations going in. We are using the rolling indices technique, and our largest indexes are of about 30GB in size (times the number of replicas for each), having 1 shard per index. The shards per node won't really help us. We also notice indices of such sizes take a lot of time to be initialized once the node has been restarted. I wonder if there's any way of speeding this up? On Sat, Jul 27, 2013 at 12:18 AM, ppearcy [ppearcy@gmail.com](mailto:ppearcy@gmail.com) wrote: > I think you need to ensure that larger indexes have more shards, so the > data can be more easily distributed. > > If you are running 0.90+ it should automatically ensure that shards for an > index are evenly (for the most part distributed): > [Add new ShardsAllocator that takes indices into account when allocating and balancing shards · Issue #2555 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2555) > > If that doesn't work for you, you can manually force max N shards per node > with this setting: > index.routing.allocation.total\_shards\_per\_node > > Best Regards, > Paul > > On Friday, July 26, 2013 10:44:16 AM UTC-6, Ivan Brusic wrote: > > > Merging in Lucene can take up a considerable amount of disk space for > > intermediate segments depending on the merge settings. It would be hard to > > take disk size into consideration for allocation without thinking of the > > merge policy as well. I can see this issue being exacerbated with > > single-shard indices (which luckily I do not have). > > > > -- > > Ivan > > > > On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko [ita...@code972.com](mailto:ita...@code972.com)wrote: > > > > > No, not sure what issue that would be. Also haven't heard back on this > > > thread or on other issues I've opened, so.... > > > > > > On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic \<[otis.gos...@gmail.com](mailto:otis.gos...@gmail.com) > > > > > > > wrote: > > > > > > > Hi Itamar, > > > > > > > > Sounds scary... did you open an issue we could watch? > > > > > > > > ## Otis > > > > > > > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/)\*\* > > > > index.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > > > On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > > > > > Unfortunately we are seeing very undesired behavior - I do hope > > > > > Elasticsearch developers will act quickly on this as this seems like a huge > > > > > show stopper > > > > > > > > > > Yes, Elasticsearch will not lose data (at least no data loss that we > > > > > were able to observe), but the current shard allocation behavior has some > > > > > very undesirable effects > > > > > > > > > > We have couple of dozen of nodes, and whenever we add more nodes the > > > > > cluster tries to rebalance. Whenever this occurs, it is very easy to notice > > > > > a very flawed balancing function is used, and it takes a couple of manual > > > > > shard allocation commands to stabilize the cluster again. > > > > > > > > > > We have some 400+ indexes with one shard each and some with multiple > > > > > replicas, some very big (10gb at least in size), some medium (around 5gb) > > > > > and many very small ones. Apparently the sharding function tries to balance > > > > > the cluster based of number of indexes on a server, without taking into > > > > > account its size on disk or number of documents in it. > > > > > > > > > > I has come to a point where we get to 0kb free on some of the nodes > > > > > during normal operation and ES unable to flush to disk. This is pretty much > > > > > unacceptable, considering we have nodes with 150gb+ free in the same > > > > > cluster. > > > > > > > > > > At the very least nodes should refuse to accept new shards if they > > > > > can't fit them in the FS taking into account other allocated (but not yet > > > > > assigned) shards > > > > > > > > > > On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann \<[v.kiss...@gmail.com](mailto:v.kiss...@gmail.com) > > > > > > > > > > > wrote: > > > > > > > > > > > Hi Otis, > > > > > > i meant both with "excellent". Excellent for us, because our data > > > > > > was still there. Not excellent for our AWS bill, we had an traffic over > > > > > > 100TB in a couple of days. > > > > > > The shards were really bounced around the cluster because every node > > > > > > was running out of disk space. So ES tried to relocate the shards all the > > > > > > time. > > > > > > Cheers, > > > > > > Vadim > > > > > > > > > > > > Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic: > > > > > > > > > > > > > Hi Vadim, > > > > > > > > > > > > > > I can't tell whether "excellent" is meant as emphasis or sarcasm? > > > > > > > > > > > > > > What do you mean by "full index was help in network (rebalancing)"? > > > > > > > It almost sounds like you are saying shards were being bounced around the > > > > > > > cluster, but maybe you meant something else? > > > > > > > > > > > > > > ## Otis > > > > > > > > > > > > > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/)\*\* > > > > > > > inde**x.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > > > > Search Analytics - [http://sematext.com/search-\*\*a](http://sematext.com/search-**a)** > > > > > > > nalytics/index.html[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > > > > > > > > > On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann > > > > > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > because ES relocate shards when your disk is full automatically. We > > > > > > > > monitored an "excellent" behavior of ES last time, than all our AWS nodes > > > > > > > > are running out of disk space. The full index was held in > > > > > > > > network(rebalancing) till a new node was joined the cluster. > > > > > > > > Cheers, > > > > > > > > Vadim > > > > > > > > > > > > > > > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar > > > > > > > > Syn-Hershko: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > I'm aware of [https://github.com/sonian/\*\*e](https://github.com/sonian/**e)\*\*\*\* > > > > > > > > > lasticsearch-equilibrium[https://github.com/sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > > > > > > > > > > > > > > > But is there a reason why similar functionality isn't implemented > > > > > > > > > in Elasticsearch itself? > > > > > > > > > > > > > > > > > > In particular, ES will try and allocate shards on machines which > > > > > > > > > are low on disk. We are seeing this all the time and I have to relocate > > > > > > > > > shards manually to prevent that. Why isn't there code in place to prevent > > > > > > > > > that? > > > > > > > > > > > > > > > > > > Itamar. > > > > > > > > > > > > > > > > -- > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > Groups "elasticsearch" group. > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > . > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > . > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![simonw\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simonw_2/32/1130_2.png) [@simonw\_2](https://discuss.elastic.co/u/simonw_2)\ **Post date:** [August 5, 2013, 9:58pm UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/11 "2013-08-05T21:58:43Z")
Hey Itamar, I think you are absolutely right that we need to add some notion of disk size / capacity etc. Regarding your question why this has not yet been implemented, well I can only say that we are planning to expand the shard allocation algorithm in the future and I am pretty sure that disk size / capacity will play a major role here. I can imagine some sort of disk space allocation decider that can restrict a node from allocating any further shards given the used / free disk space and / or move shards away given a certain limit etc. We can also make allocation decision based on the size of the shards or move shards around once they fill up and we see that certain shards are much bigger than others. All of these problems we need to address and I think we should go ahead and take the input of this thread here and open an issue, would you wanna open one and we discuss it further on the issue? simon On Sunday, August 4, 2013 4:33:34 PM UTC+2, Itamar Syn-Hershko wrote: > Ivan, I hear you, but at the very least some free space calculation needs > to be in effect. For example, when ES hits a lower limit (say, 500MB of > free space) it should start relocation out or cancel relocations going in. > > We are using the rolling indices technique, and our largest indexes are of > about 30GB in size (times the number of replicas for each), having 1 shard > per index. The shards per node won't really help us. > > We also notice indices of such sizes take a lot of time to be initialized > once the node has been restarted. I wonder if there's any way of speeding > this up? > > On Sat, Jul 27, 2013 at 12:18 AM, ppearcy \<[ppe...@gmail.com](mailto:ppe...@gmail.com) \\>wrote: > > > I think you need to ensure that larger indexes have more shards, so the > > data can be more easily distributed. > > > > If you are running 0.90+ it should automatically ensure that shards for > > an index are evenly (for the most part distributed): > > [Add new ShardsAllocator that takes indices into account when allocating and balancing shards · Issue #2555 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/2555) > > > > If that doesn't work for you, you can manually force max N shards per > > node with this setting: > > index.routing.allocation.total\_shards\_per\_node > > > > Best Regards, > > Paul > > > > On Friday, July 26, 2013 10:44:16 AM UTC-6, Ivan Brusic wrote: > > > > > Merging in Lucene can take up a considerable amount of disk space for > > > intermediate segments depending on the merge settings. It would be hard to > > > take disk size into consideration for allocation without thinking of the > > > merge policy as well. I can see this issue being exacerbated with > > > single-shard indices (which luckily I do not have). > > > > > > -- > > > Ivan > > > > > > On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko [ita...@code972.com](mailto:ita...@code972.com)wrote: > > > > > > > No, not sure what issue that would be. Also haven't heard back on this > > > > thread or on other issues I've opened, so.... > > > > > > > > On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic \< > > > > [otis.gos...@gmail.com](mailto:otis.gos...@gmail.com)\> wrote: > > > > > > > > > Hi Itamar, > > > > > > > > > > Sounds scary... did you open an issue we could watch? > > > > > > > > > > ## Otis > > > > > > > > > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/)\*\* > > > > > index.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > > Search Analytics - [http://sematext.com/search-\*\*analytics/index.html](http://sematext.com/search-**analytics/index.html)[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > > > > > On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > > > > > > > Unfortunately we are seeing very undesired behavior - I do hope > > > > > > Elasticsearch developers will act quickly on this as this seems like a huge > > > > > > show stopper > > > > > > > > > > > > Yes, Elasticsearch will not lose data (at least no data loss that we > > > > > > were able to observe), but the current shard allocation behavior has some > > > > > > very undesirable effects > > > > > > > > > > > > We have couple of dozen of nodes, and whenever we add more nodes the > > > > > > cluster tries to rebalance. Whenever this occurs, it is very easy to notice > > > > > > a very flawed balancing function is used, and it takes a couple of manual > > > > > > shard allocation commands to stabilize the cluster again. > > > > > > > > > > > > We have some 400+ indexes with one shard each and some with multiple > > > > > > replicas, some very big (10gb at least in size), some medium (around 5gb) > > > > > > and many very small ones. Apparently the sharding function tries to balance > > > > > > the cluster based of number of indexes on a server, without taking into > > > > > > account its size on disk or number of documents in it. > > > > > > > > > > > > I has come to a point where we get to 0kb free on some of the nodes > > > > > > during normal operation and ES unable to flush to disk. This is pretty much > > > > > > unacceptable, considering we have nodes with 150gb+ free in the same > > > > > > cluster. > > > > > > > > > > > > At the very least nodes should refuse to accept new shards if they > > > > > > can't fit them in the FS taking into account other allocated (but not yet > > > > > > assigned) shards > > > > > > > > > > > > On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann \< > > > > > > [v.kiss...@gmail.com](mailto:v.kiss...@gmail.com)\> wrote: > > > > > > > > > > > > > Hi Otis, > > > > > > > i meant both with "excellent". Excellent for us, because our data > > > > > > > was still there. Not excellent for our AWS bill, we had an traffic over > > > > > > > 100TB in a couple of days. > > > > > > > The shards were really bounced around the cluster because every node > > > > > > > was running out of disk space. So ES tried to relocate the shards all the > > > > > > > time. > > > > > > > Cheers, > > > > > > > Vadim > > > > > > > > > > > > > > Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic: > > > > > > > > > > > > > > > Hi Vadim, > > > > > > > > > > > > > > > > I can't tell whether "excellent" is meant as emphasis or sarcasm? > > > > > > > > > > > > > > > > What do you mean by "full index was help in network (rebalancing)"? > > > > > > > > It almost sounds like you are saying shards were being bounced around the > > > > > > > > cluster, but maybe you meant something else? > > > > > > > > > > > > > > > > ## Otis > > > > > > > > > > > > > > > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/)\*\* > > > > > > > > inde**x.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > > > > > Search Analytics - [http://sematext.com/search-\*\*a](http://sematext.com/search-**a)** > > > > > > > > nalytics/index.html[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > > > > > > > > > > > On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > because ES relocate shards when your disk is full automatically. > > > > > > > > > We monitored an "excellent" behavior of ES last time, than all our AWS > > > > > > > > > nodes are running out of disk space. The full index was held in > > > > > > > > > network(rebalancing) till a new node was joined the cluster. > > > > > > > > > Cheers, > > > > > > > > > Vadim > > > > > > > > > > > > > > > > > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar > > > > > > > > > Syn-Hershko: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > I'm aware of [https://github.com/sonian/\*\*e](https://github.com/sonian/**e)\*\*\*\* > > > > > > > > > > lasticsearch-equilibrium[https://github.com/sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > > > > > > > > > > > > > > > > > But is there a reason why similar functionality isn't > > > > > > > > > > implemented in Elasticsearch itself? > > > > > > > > > > > > > > > > > > > > In particular, ES will try and allocate shards on machines which > > > > > > > > > > are low on disk. We are seeing this all the time and I have to relocate > > > > > > > > > > shards manually to prevent that. Why isn't there code in place to prevent > > > > > > > > > > that? > > > > > > > > > > > > > > > > > > > > Itamar. > > > > > > > > > > > > > > > > > > -- > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > > . > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > . > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Itamar\_Syn\_Hershko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/itamar_syn_hershko/32/725_2.png) [@Itamar\_Syn\_Hershko](https://discuss.elastic.co/u/Itamar_Syn_Hershko)\ **Post date:** [August 10, 2013, 11:22pm UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/12 "2013-08-10T23:22:40Z")
> > > Simon says: > > \> I can imagine some sort of disk space allocation decider that can… restrict a node from > \> allocating any further shards given the used / free disk space and / or move shards > \> away given a certain limit etc. We can also make allocation decision based on the size > \> of the shards or move shards around once they fill up and we see that certain > \> shards are much bigger than others > > More at https://groups.google.com/forum/#!topic/elasticsearch/p-et4UxvcyU On Tue, Aug 6, 2013 at 12:58 AM, simonw [simon.willnauer@elasticsearch.com](mailto:simon.willnauer@elasticsearch.com)wrote: > Hey Itamar, > > I think you are absolutely right that we need to add some notion of disk > size / capacity etc. Regarding your question why this has not yet been > implemented, well I can only say that we are planning to expand the shard > allocation algorithm in the future and I am pretty sure that disk size / > capacity will play a major role here. I can imagine some sort of disk space > allocation decider that can restrict a node from allocating any further > shards given the used / free disk space and / or move shards away given a > certain limit etc. We can also make allocation decision based on the size > of the shards or move shards around once they fill up and we see that > certain shards are much bigger than others. All of these problems we need > to address and I think we should go ahead and take the input of this thread > here and open an issue, would you wanna open one and we discuss it further > on the issue? > > simon > > On Sunday, August 4, 2013 4:33:34 PM UTC+2, Itamar Syn-Hershko wrote: > > > Ivan, I hear you, but at the very least some free space calculation needs > > to be in effect. For example, when ES hits a lower limit (say, 500MB of > > free space) it should start relocation out or cancel relocations going in. > > > > We are using the rolling indices technique, and our largest indexes are > > of about 30GB in size (times the number of replicas for each), having 1 > > shard per index. The shards per node won't really help us. > > > > We also notice indices of such sizes take a lot of time to be initialized > > once the node has been restarted. I wonder if there's any way of speeding > > this up? > > > > On Sat, Jul 27, 2013 at 12:18 AM, ppearcy [ppe...@gmail.com](mailto:ppe...@gmail.com) wrote: > > > > > I think you need to ensure that larger indexes have more shards, so the > > > data can be more easily distributed. > > > > > > If you are running 0.90+ it should automatically ensure that shards for > > > an index are evenly (for the most part distributed): > > > [https://github.com/\*\*elasticsearch/elasticsearch/\*\*issues/2555](https://github.com/ **elasticsearch/elasticsearch/** issues/2555)[https://github.com/elasticsearch/elasticsearch/issues/2555](https://github.com/elasticsearch/elasticsearch/issues/2555) > > > > > > If that doesn't work for you, you can manually force max N shards per > > > node with this setting: > > > index.routing.allocation.\*\*total\_shards\_per\_node > > > > > > Best Regards, > > > Paul > > > > > > On Friday, July 26, 2013 10:44:16 AM UTC-6, Ivan Brusic wrote: > > > > > > > Merging in Lucene can take up a considerable amount of disk space for > > > > intermediate segments depending on the merge settings. It would be hard to > > > > take disk size into consideration for allocation without thinking of the > > > > merge policy as well. I can see this issue being exacerbated with > > > > single-shard indices (which luckily I do not have). > > > > > > > > -- > > > > Ivan > > > > > > > > On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko \<[ita...@code972.com](mailto:ita...@code972.com) > > > > > > > > > wrote: > > > > > > > > > No, not sure what issue that would be. Also haven't heard back on this > > > > > thread or on other issues I've opened, so.... > > > > > > > > > > On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic \< > > > > > [otis.gos...@gmail.com](mailto:otis.gos...@gmail.com)\> wrote: > > > > > > > > > > > Hi Itamar, > > > > > > > > > > > > Sounds scary... did you open an issue we could watch? > > > > > > > > > > > > ## Otis > > > > > > > > > > > > ELASTICSEARCH Performance Monitoring - [http://sematext.com/spm/\*\*inde](http://sematext.com/spm/**inde) > > > > > > **x.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > > > Search Analytics - [http://sematext.com/search-\*\*a](http://sematext.com/search-**a)** > > > > > > nalytics/index.html [http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > > > > > > > On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote: > > > > > > > > > > > > > Unfortunately we are seeing very undesired behavior - I do hope > > > > > > > Elasticsearch developers will act quickly on this as this seems like a huge > > > > > > > show stopper > > > > > > > > > > > > > > Yes, Elasticsearch will not lose data (at least no data loss that we > > > > > > > were able to observe), but the current shard allocation behavior has some > > > > > > > very undesirable effects > > > > > > > > > > > > > > We have couple of dozen of nodes, and whenever we add more nodes the > > > > > > > cluster tries to rebalance. Whenever this occurs, it is very easy to notice > > > > > > > a very flawed balancing function is used, and it takes a couple of manual > > > > > > > shard allocation commands to stabilize the cluster again. > > > > > > > > > > > > > > We have some 400+ indexes with one shard each and some with multiple > > > > > > > replicas, some very big (10gb at least in size), some medium (around 5gb) > > > > > > > and many very small ones. Apparently the sharding function tries to balance > > > > > > > the cluster based of number of indexes on a server, without taking into > > > > > > > account its size on disk or number of documents in it. > > > > > > > > > > > > > > I has come to a point where we get to 0kb free on some of the nodes > > > > > > > during normal operation and ES unable to flush to disk. This is pretty much > > > > > > > unacceptable, considering we have nodes with 150gb+ free in the same > > > > > > > cluster. > > > > > > > > > > > > > > At the very least nodes should refuse to accept new shards if they > > > > > > > can't fit them in the FS taking into account other allocated (but not yet > > > > > > > assigned) shards > > > > > > > > > > > > > > On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann \< > > > > > > > [v.kiss...@gmail.com](mailto:v.kiss...@gmail.com)\> wrote: > > > > > > > > > > > > > > > Hi Otis, > > > > > > > > i meant both with "excellent". Excellent for us, because our data > > > > > > > > was still there. Not excellent for our AWS bill, we had an traffic over > > > > > > > > 100TB in a couple of days. > > > > > > > > The shards were really bounced around the cluster because every > > > > > > > > node was running out of disk space. So ES tried to relocate the shards all > > > > > > > > the time. > > > > > > > > Cheers, > > > > > > > > Vadim > > > > > > > > > > > > > > > > Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis > > > > > > > > Gospodnetic: > > > > > > > > > > > > > > > > > Hi Vadim, > > > > > > > > > > > > > > > > > > I can't tell whether "excellent" is meant as emphasis or sarcasm? > > > > > > > > > > > > > > > > > > What do you mean by "full index was help in network > > > > > > > > > (rebalancing)"? It almost sounds like you are saying shards were being > > > > > > > > > bounced around the cluster, but maybe you meant something else? > > > > > > > > > > > > > > > > > > ## Otis > > > > > > > > > > > > > > > > > > ELASTICSEARCH Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/)\*\* > > > > > > > > > inde**x.html [http://sematext.com/spm/index.html](http://sematext.com/spm/index.html) > > > > > > > > > Search Analytics - [http://sematext.com/search-\*\*a](http://sematext.com/search-**a)** > > > > > > > > > nalytics/index.html[http://sematext.com/search-analytics/index.html](http://sematext.com/search-analytics/index.html) > > > > > > > > > > > > > > > > > > On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > because ES relocate shards when your disk is full automatically. > > > > > > > > > > We monitored an "excellent" behavior of ES last time, than all our AWS > > > > > > > > > > nodes are running out of disk space. The full index was held in > > > > > > > > > > network(rebalancing) till a new node was joined the cluster. > > > > > > > > > > Cheers, > > > > > > > > > > Vadim > > > > > > > > > > > > > > > > > > > > Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar > > > > > > > > > > Syn-Hershko: > > > > > > > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > > > > > > > I'm aware of [https://github.com/sonian/\*\*e](https://github.com/sonian/**e)\*\*\*\*\*\* > > > > > > > > > > > lasticsearch-equilibrium[https://github.com/sonian/elasticsearch-equilibrium](https://github.com/sonian/elasticsearch-equilibrium) > > > > > > > > > > > > > > > > > > > > > > But is there a reason why similar functionality isn't > > > > > > > > > > > implemented in Elasticsearch itself? > > > > > > > > > > > > > > > > > > > > > > In particular, ES will try and allocate shards on machines which > > > > > > > > > > > are low on disk. We are seeing this all the time and I have to relocate > > > > > > > > > > > shards manually to prevent that. Why isn't there code in place to prevent > > > > > > > > > > > that? > > > > > > > > > > > > > > > > > > > > > > Itamar. > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > You received this message because you are subscribed to the Google > > > > > > > > > > Groups "elasticsearch" group. > > > > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > > > > send an email to elasticsearc...@\*\*googlegroups. **c** om. > > > > > > > > > > > > > > > > For more options, visit [https://groups.google.com/\*\*grou](https://groups.google.com/**grou)\*\*\*\* > > > > > > > > ps/opt\_out [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > > > > > > > > > -- > > > > > > > You received this message because you are subscribed to the Google > > > > > > > Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > > send an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > > > . > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to elasticsearc...@\*\*googlegroups.\*\*com. > > > > > For more options, visit [https://groups.google.com/\*\*grou\*\*ps/opt\_out](https://groups.google.com/ **grou** ps/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > > . > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > > . > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:21am UTC](https://discuss.elastic.co/t/shard-allocation-logic-not-taking-disk-size-into-account-why/12743/13 "2017-07-06T02:21:43Z")