# Sharding unbalance problem **URL:** \ **Category:** Elasticsearch\ **Created:** [September 30, 2012, 1:57am UTC](https://discuss.elastic.co/t/sharding-unbalance-problem/9194 "2012-09-30T01:57:52Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Jae](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jae/32/791_2.png) [@Jae](https://discuss.elastic.co/u/Jae)\ **Post date:** [September 30, 2012, 1:57am UTC](https://discuss.elastic.co/t/sharding-unbalance-problem/9194/1 "2012-09-30T01:57:52Z")
Hi Today, my elasticsearch 0.19.9 cluster was down because of 'no free space'. I am seeing shards are not allocated with balance. My question is how to monitor free space of each instance and what the best method is to prevent 'no free space', just deleting old index frequently and checking free space in each instance? When I chekced the free space percentage using 'df -H' command, many instances were showing less than 70% disk usage but one instance got no empty space. Please give me the guideline regarding free space monitoring. Thank you Best, Jae -- ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [October 1, 2012, 5:10pm UTC](https://discuss.elastic.co/t/sharding-unbalance-problem/9194/2 "2012-10-01T17:10:58Z")
Are you nodes not equal in terms of disk capacity? Do you have different indices with different shard counts? Elasticsearch assumes every node in the cluster are heterogeneous. Shard allocation is based purely on placing shards in an even distribution among the nodes. The size of the disk, the number of shards per index, and the number of indices do not matter. If one index's shard size is bigger than another index's shard size, then the shard allocator could potentially put the more heavyweight shards together on one machine. Elasticsearch does not provide any monitoring tools that provide notification (perhaps SPM does?). You should setup monitoring software on your machines for OS level stats (CPU/mem/disk). Cheers, Ivan On Sat, Sep 29, 2012 at 6:57 PM, Jae [metacret@gmail.com](mailto:metacret@gmail.com) wrote: > Hi > > Today, my elasticsearch 0.19.9 cluster was down because of 'no free > space'. I am seeing shards are not allocated with balance. My question is > how to monitor free space of each instance and what the best method is to > prevent 'no free space', just deleting old index frequently and checking > free space in each instance? > > When I chekced the free space percentage using 'df -H' command, many > instances were showing less than 70% disk usage but one instance got no > empty space. Please give me the guideline regarding free space monitoring. > > Thank you > Best, Jae > > -- -- ---
**Author:** ![Jae](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jae/32/791_2.png) [@Jae](https://discuss.elastic.co/u/Jae)\ **Post date:** [October 1, 2012, 5:29pm UTC](https://discuss.elastic.co/t/sharding-unbalance-problem/9194/3 "2012-10-01T17:29:17Z")
On Monday, October 1, 2012 10:11:03 AM UTC-7, Ivan Brusic wrote: > Are you nodes not equal in terms of disk capacity? Do you have different > indices with different shard counts? No, every node has equal disk space. Every index has the same shard count and replication count. I deleted several indices but only one instance is consuming 68%. The following is the result of 'df -H' for each instance. FYI, I am using TransportClient and sample and ping interval is 60s. i-1c797166 /dev/md0 1.9T 939G 865G 53% /mnt i-12797168 /dev/md0 1.9T 938G 867G 52% /mnt i-1079716a /dev/md0 1.9T 812G 992G 46% /mnt i-1679716c /dev/md0 1.9T 938G 867G 52% /mnt i-7c7c7406 /dev/md0 1.9T 813G 992G 46% /mnt i-727c7408 /dev/md0 1.9T 839G 966G 47% /mnt i-767c740c /dev/md0 1.9T 1.3T 591G 68% /mnt i-747c740e /dev/md0 1.9T 1.1T 740G 60% /mnt i-327f7748 /dev/md0 1.9T 812G 992G 46% /mnt i-307f774a /dev/md0 1.9T 1.1T 740G 60% /mnt i-347f774e /dev/md0 1.9T 1.1T 740G 59% /mnt i-26747c5c /dev/md0 1.9T 938G 866G 53% /mnt i-08747c72 /dev/md0 1.9T 1.1T 740G 59% /mnt i-0e747c74 /dev/md0 1.9T 813G 992G 46% /mnt i-327f7748 /dev/md0 1.9T 812G 992G 46% /mnt i-307f774a /dev/md0 1.9T 1.1T 740G 60% /mnt i-347f774e /dev/md0 1.9T 1.1T 740G 59% /mnt i-26747c5c /dev/md0 1.9T 938G 866G 53% /mnt i-08747c72 /dev/md0 1.9T 1.1T 740G 59% /mnt i-0e747c74 /dev/md0 1.9T 813G 992G 46% /mnt i-04747c7e /dev/md0 1.9T 813G 992G 46% /mnt i-ea767e90 /dev/md0 1.9T 938G 866G 52% /mnt i-e8767e92 /dev/md0 1.9T 839G 966G 47% /mnt i-ec767e96 /dev/md0 1.9T 812G 992G 46% /mnt i-e2767e98 /dev/md0 1.9T 938G 866G 53% /mnt i-2a080050 /dev/md0 1.9T 1.1T 740G 59% /mnt i-2e080054 /dev/md0 1.9T 1.1T 740G 59% /mnt i-2c080056 /dev/md0 1.9T 939G 866G 53% /mnt i-2608005c /dev/md0 1.9T 1.2T 641G 65% /mnt i-f40a028e /dev/md0 1.9T 1.1T 740G 59% /mnt i-ea0a0290 /dev/md0 1.9T 938G 866G 53% /mnt i-ee0a0294 /dev/md0 1.9T 1.1T 740G 59% /mnt i-360d054c /dev/md0 1.9T 938G 866G 53% /mnt i-340d054e /dev/md0 1.9T 938G 866G 53% /mnt i-280d0552 /dev/md0 1.9T 813G 992G 46% /mnt i-2e0d0554 /dev/md0 1.9T 938G 866G 53% /mnt i-720f0708 /dev/md0 1.9T 812G 992G 46% /mnt i-700f070a /dev/md0 1.9T 939G 866G 53% /mnt i-6c0f0716 /dev/md0 1.9T 938G 866G 52% /mnt i-620f0718 /dev/md0 1.9T 764G 1.1T 43% /mnt i-44f8ec3e /dev/md0 1.9T 939G 866G 53% /mnt i-838a79fe /dev/md0 1.9T 813G 992G 46% /mnt Thank you Best, Jae > Elasticsearch assumes every node in the cluster are heterogeneous. Shard > allocation is based purely on placing shards in an even distribution among > the nodes. The size of the disk, the number of shards per index, and the > number of indices do not matter. If one index's shard size is bigger than > another index's shard size, then the shard allocator could potentially put > the more heavyweight shards together on one machine. > > Elasticsearch does not provide any monitoring tools that provide > notification (perhaps SPM does?). You should setup monitoring software on > your machines for OS level stats (CPU/mem/disk). > > Cheers, > > Ivan > > On Sat, Sep 29, 2012 at 6:57 PM, Jae \<[meta...@gmail.com](mailto:meta...@gmail.com) \\>wrote: > > > Hi > > > > Today, my elasticsearch 0.19.9 cluster was down because of 'no free > > space'. I am seeing shards are not allocated with balance. My question is > > how to monitor free space of each instance and what the best method is to > > prevent 'no free space', just deleting old index frequently and checking > > free space in each instance? > > > > When I chekced the free space percentage using 'df -H' command, many > > instances were showing less than 70% disk usage but one instance got no > > empty space. Please give me the guideline regarding free space monitoring. > > > > Thank you > > Best, Jae > > > > -- -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [October 2, 2012, 12:55am UTC](https://discuss.elastic.co/t/sharding-unbalance-problem/9194/4 "2012-10-02T00:55:33Z")
Hello, Correct, if you use SPM for ES (or any other flavour of SPM for that matter) you can monitor the disk space and set an alert when its usage reaches N%, say 90%, so you can be automatically notified instead of having to proactively watch disk space. For more info about SPM, check the URL in the sig or email off-list. On the ES side, the new ES has a mechanism to manually moved shards around the cluster, which may be handy for you here. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Monday, October 1, 2012 1:11:03 PM UTC-4, Ivan Brusic wrote: > Are you nodes not equal in terms of disk capacity? Do you have different > indices with different shard counts? > > Elasticsearch assumes every node in the cluster are heterogeneous. Shard > allocation is based purely on placing shards in an even distribution among > the nodes. The size of the disk, the number of shards per index, and the > number of indices do not matter. If one index's shard size is bigger than > another index's shard size, then the shard allocator could potentially put > the more heavyweight shards together on one machine. > > Elasticsearch does not provide any monitoring tools that provide > notification (perhaps SPM does?). You should setup monitoring software on > your machines for OS level stats (CPU/mem/disk). > > Cheers, > > Ivan > > On Sat, Sep 29, 2012 at 6:57 PM, Jae \<[meta...@gmail.com](mailto:meta...@gmail.com) \\>wrote: > > > Hi > > > > Today, my elasticsearch 0.19.9 cluster was down because of 'no free > > space'. I am seeing shards are not allocated with balance. My question is > > how to monitor free space of each instance and what the best method is to > > prevent 'no free space', just deleting old index frequently and checking > > free space in each instance? > > > > When I chekced the free space percentage using 'df -H' command, many > > instances were showing less than 70% disk usage but one instance got no > > empty space. Please give me the guideline regarding free space monitoring. > > > > Thank you > > Best, Jae > > > > -- -- ---
**Author:** ![phill](https://avatars.discourse-cdn.com/v4/letter/p/779978/32.png) [@phill](https://discuss.elastic.co/u/phill)\ **Post date:** [October 3, 2012, 9:05pm UTC](https://discuss.elastic.co/t/sharding-unbalance-problem/9194/5 "2012-10-03T21:05:40Z")
On 10/1/2012 5:55 PM, Otis Gospodnetic wrote: > On the ES side, the new ES has a mechanism to manually moved shards > around the cluster, which may be handy for you here. New? What version? This sounds interesting. Is it something in the index "module" realm? -Paul -- ---
**Author:** ![Paul\_Smith](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_smith/32/1323_2.png) [@Paul\_Smith](https://discuss.elastic.co/u/Paul_Smith)\ **Post date:** [October 3, 2012, 9:22pm UTC](https://discuss.elastic.co/t/sharding-unbalance-problem/9194/6 "2012-10-03T21:22:25Z")
0.19.10 released recently has the Cluster Reroute API added: > > > The reroute command allows to explcitiyly execute a cluster reroute allocation c…ommand including specific commands. For example, a shard can be moved from one node to another explicitly, an allocation can be canceled, or an unassigned shard can be explicitly allocated on a specific node. > > Here is a short example of how a simple reroute API call: > > \`\`\` > curl -XPOST 'localhost:9200/\_cluster/reroute' -d '{ > "commands" : \[ > {"move" : {"index" : "test", "shard" : 0, "from\_node" : "node1", "to\_node" : "node2"}}, > {"allocate" : {"index" : "test", "shard" : 1, "node" : "node3"}} > \] > }' > \`\`\` > > An importnat aspect to remember is the fact that once when an allocation occurs, the cluster will aim at rebalancing its state back to an even state. For example, if the allocation includes moving a shard from \`node1\` to \`node2\`, in an "even" state, then another shard will be moved from \`node2\` to \`node1\` to even things out. > > The cluster can be set to disable allocations, which means that only the explicitl allocations will be performed. Obviously, only once all commands has been applied, the cluster will aim to be rebalance its state. > > Anohter option is to run the commands in "dry\_run" (as a URI flag, or in the request body). This will cause the commands to apply to the current cluster state, and reutrn the resulting cluster after the comamnds (and rebalancing) has been applied. > > The commands supporterd are: > \- \`move\`: Move a started shard from one node to anotehr node. Accepts \`index\` and \`shard\` for index name and shard number, \`from\_node\` for the node to move the shard "from", and \`to\_node\` for the node to move the shard to. > \- \`cancel\`: Cancel allocation of a shard (or recovery). Accepts \`index\` and \`shard\` for index name and shar number, and \`node\` for the node to cancel the shard allocation on. It also accepts \`allow\_primary\` flag to explciitly specify that it is allowed to cancel allocation for a primary shard. > \- \`allocate\`: Allocate an unassigned shard to a node. Accepts the \`index\` and \`shard\` for index name and shard number, and \`node\` to allocate the shard to. It also accepts \`allow\_primary\` flag to explciitly specify that it is allowed to explciitly allocate a primary shard (might result in data loss). On 4 October 2012 07:07, P. Hill [parehill1@gmail.com](mailto:parehill1@gmail.com) wrote: > On 10/1/2012 5:55 PM, Otis Gospodnetic wrote: > > > On the ES side, the new ES has a mechanism to manually moved shards > > around the cluster, which may be handy for you here. > > > > New? What version? This sounds interesting. Is it something in the > > index "module" realm? > > -Paul > > -- -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:10am UTC](https://discuss.elastic.co/t/sharding-unbalance-problem/9194/7 "2017-07-06T03:10:16Z")