Sharding unbalance problem

Jae · September 30, 2012, 1:57am

Hi

Today, my elasticsearch 0.19.9 cluster was down because of 'no free space'.
I am seeing shards are not allocated with balance. My question is how to
monitor free space of each instance and what the best method is to prevent
'no free space', just deleting old index frequently and checking free space
in each instance?

When I chekced the free space percentage using 'df -H' command, many
instances were showing less than 70% disk usage but one instance got no
empty space. Please give me the guideline regarding free space monitoring.

Thank you
Best, Jae

--

Ivan · October 1, 2012, 5:10pm

Are you nodes not equal in terms of disk capacity? Do you have different
indices with different shard counts?

Elasticsearch assumes every node in the cluster are heterogeneous. Shard
allocation is based purely on placing shards in an even distribution among
the nodes. The size of the disk, the number of shards per index, and the
number of indices do not matter. If one index's shard size is bigger than
another index's shard size, then the shard allocator could potentially put
the more heavyweight shards together on one machine.

Elasticsearch does not provide any monitoring tools that provide
notification (perhaps SPM does?). You should setup monitoring software on
your machines for OS level stats (CPU/mem/disk).

Cheers,

Ivan

On Sat, Sep 29, 2012 at 6:57 PM, Jae metacret@gmail.com wrote:

Hi

Today, my elasticsearch 0.19.9 cluster was down because of 'no free
space'. I am seeing shards are not allocated with balance. My question is
how to monitor free space of each instance and what the best method is to
prevent 'no free space', just deleting old index frequently and checking
free space in each instance?

When I chekced the free space percentage using 'df -H' command, many
instances were showing less than 70% disk usage but one instance got no
empty space. Please give me the guideline regarding free space monitoring.

Thank you
Best, Jae

--

--

Jae · October 1, 2012, 5:29pm

On Monday, October 1, 2012 10:11:03 AM UTC-7, Ivan Brusic wrote:

Are you nodes not equal in terms of disk capacity? Do you have different
indices with different shard counts?

No, every node has equal disk space. Every index has the same shard count
and replication count.

I deleted several indices but only one instance is consuming 68%. The
following is the result of 'df -H' for each instance.

FYI, I am using TransportClient and sample and ping interval is 60s.

i-1c797166
/dev/md0 1.9T 939G 865G 53% /mnt
i-12797168
/dev/md0 1.9T 938G 867G 52% /mnt
i-1079716a
/dev/md0 1.9T 812G 992G 46% /mnt
i-1679716c
/dev/md0 1.9T 938G 867G 52% /mnt
i-7c7c7406
/dev/md0 1.9T 813G 992G 46% /mnt
i-727c7408
/dev/md0 1.9T 839G 966G 47% /mnt
i-767c740c
/dev/md0 1.9T 1.3T 591G 68% /mnt
i-747c740e
/dev/md0 1.9T 1.1T 740G 60% /mnt
i-327f7748
/dev/md0 1.9T 812G 992G 46% /mnt
i-307f774a
/dev/md0 1.9T 1.1T 740G 60% /mnt
i-347f774e
/dev/md0 1.9T 1.1T 740G 59% /mnt
i-26747c5c
/dev/md0 1.9T 938G 866G 53% /mnt
i-08747c72
/dev/md0 1.9T 1.1T 740G 59% /mnt
i-0e747c74
/dev/md0 1.9T 813G 992G 46% /mnt
i-327f7748
/dev/md0 1.9T 812G 992G 46% /mnt
i-307f774a
/dev/md0 1.9T 1.1T 740G 60% /mnt
i-347f774e
/dev/md0 1.9T 1.1T 740G 59% /mnt
i-26747c5c
/dev/md0 1.9T 938G 866G 53% /mnt
i-08747c72
/dev/md0 1.9T 1.1T 740G 59% /mnt
i-0e747c74
/dev/md0 1.9T 813G 992G 46% /mnt
i-04747c7e
/dev/md0 1.9T 813G 992G 46% /mnt
i-ea767e90
/dev/md0 1.9T 938G 866G 52% /mnt
i-e8767e92
/dev/md0 1.9T 839G 966G 47% /mnt
i-ec767e96
/dev/md0 1.9T 812G 992G 46% /mnt
i-e2767e98
/dev/md0 1.9T 938G 866G 53% /mnt
i-2a080050
/dev/md0 1.9T 1.1T 740G 59% /mnt
i-2e080054
/dev/md0 1.9T 1.1T 740G 59% /mnt
i-2c080056
/dev/md0 1.9T 939G 866G 53% /mnt
i-2608005c
/dev/md0 1.9T 1.2T 641G 65% /mnt
i-f40a028e
/dev/md0 1.9T 1.1T 740G 59% /mnt
i-ea0a0290
/dev/md0 1.9T 938G 866G 53% /mnt
i-ee0a0294
/dev/md0 1.9T 1.1T 740G 59% /mnt
i-360d054c
/dev/md0 1.9T 938G 866G 53% /mnt
i-340d054e
/dev/md0 1.9T 938G 866G 53% /mnt
i-280d0552
/dev/md0 1.9T 813G 992G 46% /mnt
i-2e0d0554
/dev/md0 1.9T 938G 866G 53% /mnt
i-720f0708
/dev/md0 1.9T 812G 992G 46% /mnt
i-700f070a
/dev/md0 1.9T 939G 866G 53% /mnt
i-6c0f0716
/dev/md0 1.9T 938G 866G 52% /mnt
i-620f0718
/dev/md0 1.9T 764G 1.1T 43% /mnt
i-44f8ec3e
/dev/md0 1.9T 939G 866G 53% /mnt
i-838a79fe
/dev/md0 1.9T 813G 992G 46% /mnt

Thank you
Best, Jae

Elasticsearch assumes every node in the cluster are heterogeneous. Shard
allocation is based purely on placing shards in an even distribution among
the nodes. The size of the disk, the number of shards per index, and the
number of indices do not matter. If one index's shard size is bigger than
another index's shard size, then the shard allocator could potentially put
the more heavyweight shards together on one machine.

Elasticsearch does not provide any monitoring tools that provide
notification (perhaps SPM does?). You should setup monitoring software on
your machines for OS level stats (CPU/mem/disk).

Cheers,

Ivan

On Sat, Sep 29, 2012 at 6:57 PM, Jae <meta...@gmail.com <javascript:>>wrote:

Hi

Today, my elasticsearch 0.19.9 cluster was down because of 'no free
space'. I am seeing shards are not allocated with balance. My question is
how to monitor free space of each instance and what the best method is to
prevent 'no free space', just deleting old index frequently and checking
free space in each instance?

When I chekced the free space percentage using 'df -H' command, many
instances were showing less than 70% disk usage but one instance got no
empty space. Please give me the guideline regarding free space monitoring.

Thank you
Best, Jae

--

--

otisg · October 2, 2012, 12:55am

Hello,

Correct, if you use SPM for ES (or any other flavour of SPM for that
matter) you can monitor the disk space and set an alert when its usage
reaches N%, say 90%, so you can be automatically notified instead of having
to proactively watch disk space. For more info about SPM, check the URL in
the sig or email off-list.

On the ES side, the new ES has a mechanism to manually moved shards around
the cluster, which may be handy for you here.

Otis

Search Analytics - Cloud Monitoring Tools & Services | Sematext
Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service

On Monday, October 1, 2012 1:11:03 PM UTC-4, Ivan Brusic wrote:

Are you nodes not equal in terms of disk capacity? Do you have different
indices with different shard counts?

Elasticsearch assumes every node in the cluster are heterogeneous. Shard
allocation is based purely on placing shards in an even distribution among
the nodes. The size of the disk, the number of shards per index, and the
number of indices do not matter. If one index's shard size is bigger than
another index's shard size, then the shard allocator could potentially put
the more heavyweight shards together on one machine.

Elasticsearch does not provide any monitoring tools that provide
notification (perhaps SPM does?). You should setup monitoring software on
your machines for OS level stats (CPU/mem/disk).

Cheers,

Ivan

On Sat, Sep 29, 2012 at 6:57 PM, Jae <meta...@gmail.com <javascript:>>wrote:

Hi

Today, my elasticsearch 0.19.9 cluster was down because of 'no free
space'. I am seeing shards are not allocated with balance. My question is
how to monitor free space of each instance and what the best method is to
prevent 'no free space', just deleting old index frequently and checking
free space in each instance?

When I chekced the free space percentage using 'df -H' command, many
instances were showing less than 70% disk usage but one instance got no
empty space. Please give me the guideline regarding free space monitoring.

Thank you
Best, Jae

--

--

phill · October 3, 2012, 9:05pm

On 10/1/2012 5:55 PM, Otis Gospodnetic wrote:

On the ES side, the new ES has a mechanism to manually moved shards
around the cluster, which may be handy for you here.

New? What version? This sounds interesting. Is it something in the
index "module" realm?

-Paul

--

Paul_Smith · October 3, 2012, 9:22pm

0.19.10 released recently has the Cluster Reroute API added:

github.com/elastic/elasticsearch

Cluster Reroute API: Allow to specify allocation commands

opened 08:55PM - 15 Sep 12 UTC

closed 08:56PM - 15 Sep 12 UTC

kimchy

>feature v0.20.0.RC1 v0.19.10

The reroute command allows to explcitiyly execute a cluster reroute allocation c…ommand including specific commands. For example, a shard can be moved from one node to another explicitly, an allocation can be canceled, or an unassigned shard can be explicitly allocated on a specific node. Here is a short example of how a simple reroute API call: ``` curl -XPOST 'localhost:9200/_cluster/reroute' -d '{ "commands" : [ {"move" : {"index" : "test", "shard" : 0, "from_node" : "node1", "to_node" : "node2"}}, {"allocate" : {"index" : "test", "shard" : 1, "node" : "node3"}} ] }' ``` An importnat aspect to remember is the fact that once when an allocation occurs, the cluster will aim at rebalancing its state back to an even state. For example, if the allocation includes moving a shard from `node1` to `node2`, in an "even" state, then another shard will be moved from `node2` to `node1` to even things out. The cluster can be set to disable allocations, which means that only the explicitl allocations will be performed. Obviously, only once all commands has been applied, the cluster will aim to be rebalance its state. Anohter option is to run the commands in "dry_run" (as a URI flag, or in the request body). This will cause the commands to apply to the current cluster state, and reutrn the resulting cluster after the comamnds (and rebalancing) has been applied. The commands supporterd are: - `move`: Move a started shard from one node to anotehr node. Accepts `index` and `shard` for index name and shard number, `from_node` for the node to move the shard "from", and `to_node` for the node to move the shard to. - `cancel`: Cancel allocation of a shard (or recovery). Accepts `index` and `shard` for index name and shar number, and `node` for the node to cancel the shard allocation on. It also accepts `allow_primary` flag to explciitly specify that it is allowed to cancel allocation for a primary shard. - `allocate`: Allocate an unassigned shard to a node. Accepts the `index` and `shard` for index name and shard number, and `node` to allocate the shard to. It also accepts `allow_primary` flag to explciitly specify that it is allowed to explciitly allocate a primary shard (might result in data loss).

On 4 October 2012 07:07, P. Hill parehill1@gmail.com wrote:

On 10/1/2012 5:55 PM, Otis Gospodnetic wrote:

On the ES side, the new ES has a mechanism to manually moved shards
around the cluster, which may be handy for you here.

New? What version? This sounds interesting. Is it something in the
index "module" realm?

-Paul

--

--