Shard allocation logic not taking disk size into account - why?

Hi,

I'm aware of https://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented in
ElasticSearch itself?

In particular, ES will try and allocate shards on machines which are low on
disk. We are seeing this all the time and I have to relocate shards
manually to prevent that. Why isn't there code in place to prevent that?

Itamar.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented in
ElasticSearch itself?

In particular, ES will try and allocate shards on machines which are low
on disk. We are seeing this all the time and I have to relocate shards
manually to prevent that. Why isn't there code in place to prevent that?

Itamar.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network (rebalancing)"? It
almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote:

Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented in
ElasticSearch itself?

In particular, ES will try and allocate shards on machines which are low
on disk. We are seeing this all the time and I have to relocate shards
manually to prevent that. Why isn't there code in place to prevent that?

Itamar.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Otis,
i meant both with "excellent". Excellent for us, because our data was
still there. Not excellent for our AWS bill, we had an traffic over 100TB
in a couple of days.
The shards were really bounced around the cluster because every node was
running out of disk space. So ES tried to relocate the shards all the time.
Cheers,
Vadim

Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic:

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network (rebalancing)"? It
almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote:

Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented in
ElasticSearch itself?

In particular, ES will try and allocate shards on machines which are low
on disk. We are seeing this all the time and I have to relocate shards
manually to prevent that. Why isn't there code in place to prevent that?

Itamar.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Unfortunately we are seeing very undesired behavior - I do hope
ElasticSearch developers will act quickly on this as this seems like a huge
show stopper

Yes, ElasticSearch will not lose data (at least no data loss that we were
able to observe), but the current shard allocation behavior has some very
undesirable effects

We have couple of dozen of nodes, and whenever we add more nodes the
cluster tries to rebalance. Whenever this occurs, it is very easy to notice
a very flawed balancing function is used, and it takes a couple of manual
shard allocation commands to stabilize the cluster again.

We have some 400+ indexes with one shard each and some with multiple
replicas, some very big (10gb at least in size), some medium (around 5gb)
and many very small ones. Apparently the sharding function tries to balance
the cluster based of number of indexes on a server, without taking into
account its size on disk or number of documents in it.

I has come to a point where we get to 0kb free on some of the nodes during
normal operation and ES unable to flush to disk. This is pretty much
unacceptable, considering we have nodes with 150gb+ free in the same
cluster.

At the very least nodes should refuse to accept new shards if they can't
fit them in the FS taking into account other allocated (but not yet
assigned) shards

On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann v.kisselmann@gmail.comwrote:

Hi Otis,
i meant both with "excellent". Excellent for us, because our data was
still there. Not excellent for our AWS bill, we had an traffic over 100TB
in a couple of days.
The shards were really bounced around the cluster because every node was
running out of disk space. So ES tried to relocate the shards all the time.
Cheers,
Vadim

Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic:

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network (rebalancing)"? It
almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**
index.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote:

Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/**elasticsearch-equilibriumhttps://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented in
ElasticSearch itself?

In particular, ES will try and allocate shards on machines which are
low on disk. We are seeing this all the time and I have to relocate shards
manually to prevent that. Why isn't there code in place to prevent that?

Itamar.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Itamar,

Sounds scary... did you open an issue we could watch?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html

On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote:

Unfortunately we are seeing very undesired behavior - I do hope
ElasticSearch developers will act quickly on this as this seems like a huge
show stopper

Yes, ElasticSearch will not lose data (at least no data loss that we were
able to observe), but the current shard allocation behavior has some very
undesirable effects

We have couple of dozen of nodes, and whenever we add more nodes the
cluster tries to rebalance. Whenever this occurs, it is very easy to notice
a very flawed balancing function is used, and it takes a couple of manual
shard allocation commands to stabilize the cluster again.

We have some 400+ indexes with one shard each and some with multiple
replicas, some very big (10gb at least in size), some medium (around 5gb)
and many very small ones. Apparently the sharding function tries to balance
the cluster based of number of indexes on a server, without taking into
account its size on disk or number of documents in it.

I has come to a point where we get to 0kb free on some of the nodes during
normal operation and ES unable to flush to disk. This is pretty much
unacceptable, considering we have nodes with 150gb+ free in the same
cluster.

At the very least nodes should refuse to accept new shards if they can't
fit them in the FS taking into account other allocated (but not yet
assigned) shards

On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann <v.kiss...@gmail.com<javascript:>

wrote:

Hi Otis,
i meant both with "excellent". Excellent for us, because our data was
still there. Not excellent for our AWS bill, we had an traffic over 100TB
in a couple of days.
The shards were really bounced around the cluster because every node was
running out of disk space. So ES tried to relocate the shards all the time.
Cheers,
Vadim

Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic:

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network (rebalancing)"? It
almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**
index.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote:

Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/**elasticsearch-equilibriumhttps://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented in
ElasticSearch itself?

In particular, ES will try and allocate shards on machines which are
low on disk. We are seeing this all the time and I have to relocate shards
manually to prevent that. Why isn't there code in place to prevent that?

Itamar.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

No, not sure what issue that would be. Also haven't heard back on this
thread or on other issues I've opened, so....

On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi Itamar,

Sounds scary... did you open an issue we could watch?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html

On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote:

Unfortunately we are seeing very undesired behavior - I do hope
ElasticSearch developers will act quickly on this as this seems like a huge
show stopper

Yes, ElasticSearch will not lose data (at least no data loss that we were
able to observe), but the current shard allocation behavior has some very
undesirable effects

We have couple of dozen of nodes, and whenever we add more nodes the
cluster tries to rebalance. Whenever this occurs, it is very easy to notice
a very flawed balancing function is used, and it takes a couple of manual
shard allocation commands to stabilize the cluster again.

We have some 400+ indexes with one shard each and some with multiple
replicas, some very big (10gb at least in size), some medium (around 5gb)
and many very small ones. Apparently the sharding function tries to balance
the cluster based of number of indexes on a server, without taking into
account its size on disk or number of documents in it.

I has come to a point where we get to 0kb free on some of the nodes
during normal operation and ES unable to flush to disk. This is pretty much
unacceptable, considering we have nodes with 150gb+ free in the same
cluster.

At the very least nodes should refuse to accept new shards if they can't
fit them in the FS taking into account other allocated (but not yet
assigned) shards

On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann v.kiss...@gmail.comwrote:

Hi Otis,
i meant both with "excellent". Excellent for us, because our data was
still there. Not excellent for our AWS bill, we had an traffic over 100TB
in a couple of days.
The shards were really bounced around the cluster because every node was
running out of disk space. So ES tried to relocate the shards all the time.
Cheers,
Vadim

Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic:

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network (rebalancing)"? It
almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/inde
x.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.htmlhttp://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote:

Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/elasticsearch-equilibriumhttps://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented in
ElasticSearch itself?

In particular, ES will try and allocate shards on machines which are
low on disk. We are seeing this all the time and I have to relocate shards
manually to prevent that. Why isn't there code in place to prevent that?

Itamar.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Merging in Lucene can take up a considerable amount of disk space for
intermediate segments depending on the merge settings. It would be hard to
take disk size into consideration for allocation without thinking of the
merge policy as well. I can see this issue being exacerbated with
single-shard indices (which luckily I do not have).

--
Ivan

On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko itamar@code972.comwrote:

No, not sure what issue that would be. Also haven't heard back on this
thread or on other issues I've opened, so....

On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi Itamar,

Sounds scary... did you open an issue we could watch?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html

On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote:

Unfortunately we are seeing very undesired behavior - I do hope
ElasticSearch developers will act quickly on this as this seems like a huge
show stopper

Yes, ElasticSearch will not lose data (at least no data loss that we
were able to observe), but the current shard allocation behavior has some
very undesirable effects

We have couple of dozen of nodes, and whenever we add more nodes the
cluster tries to rebalance. Whenever this occurs, it is very easy to notice
a very flawed balancing function is used, and it takes a couple of manual
shard allocation commands to stabilize the cluster again.

We have some 400+ indexes with one shard each and some with multiple
replicas, some very big (10gb at least in size), some medium (around 5gb)
and many very small ones. Apparently the sharding function tries to balance
the cluster based of number of indexes on a server, without taking into
account its size on disk or number of documents in it.

I has come to a point where we get to 0kb free on some of the nodes
during normal operation and ES unable to flush to disk. This is pretty much
unacceptable, considering we have nodes with 150gb+ free in the same
cluster.

At the very least nodes should refuse to accept new shards if they can't
fit them in the FS taking into account other allocated (but not yet
assigned) shards

On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann v.kiss...@gmail.comwrote:

Hi Otis,
i meant both with "excellent". Excellent for us, because our data was
still there. Not excellent for our AWS bill, we had an traffic over 100TB
in a couple of days.
The shards were really bounced around the cluster because every node
was running out of disk space. So ES tried to relocate the shards all the
time.
Cheers,
Vadim

Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic:

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network (rebalancing)"?
It almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/*inde
*x.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.htmlhttp://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote:

Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/elasticsearch-equilibriumhttps://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented
in ElasticSearch itself?

In particular, ES will try and allocate shards on machines which are
low on disk. We are seeing this all the time and I have to relocate shards
manually to prevent that. Why isn't there code in place to prevent that?

Itamar.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I think you need to ensure that larger indexes have more shards, so the
data can be more easily distributed.

If you are running 0.90+ it should automatically ensure that shards for an
index are evenly (for the most part distributed):

If that doesn't work for you, you can manually force max N shards per node
with this setting:
index.routing.allocation.total_shards_per_node

Best Regards,
Paul

On Friday, July 26, 2013 10:44:16 AM UTC-6, Ivan Brusic wrote:

Merging in Lucene can take up a considerable amount of disk space for
intermediate segments depending on the merge settings. It would be hard to
take disk size into consideration for allocation without thinking of the
merge policy as well. I can see this issue being exacerbated with
single-shard indices (which luckily I do not have).

--
Ivan

On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko <ita...@code972.com<javascript:>

wrote:

No, not sure what issue that would be. Also haven't heard back on this
thread or on other issues I've opened, so....

On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic <otis.gos...@gmail.com<javascript:>

wrote:

Hi Itamar,

Sounds scary... did you open an issue we could watch?

Otis

ELASTICSEARCH Performance Monitoring -
http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-analytics/index.html

On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote:

Unfortunately we are seeing very undesired behavior - I do hope
ElasticSearch developers will act quickly on this as this seems like a huge
show stopper

Yes, ElasticSearch will not lose data (at least no data loss that we
were able to observe), but the current shard allocation behavior has some
very undesirable effects

We have couple of dozen of nodes, and whenever we add more nodes the
cluster tries to rebalance. Whenever this occurs, it is very easy to notice
a very flawed balancing function is used, and it takes a couple of manual
shard allocation commands to stabilize the cluster again.

We have some 400+ indexes with one shard each and some with multiple
replicas, some very big (10gb at least in size), some medium (around 5gb)
and many very small ones. Apparently the sharding function tries to balance
the cluster based of number of indexes on a server, without taking into
account its size on disk or number of documents in it.

I has come to a point where we get to 0kb free on some of the nodes
during normal operation and ES unable to flush to disk. This is pretty much
unacceptable, considering we have nodes with 150gb+ free in the same
cluster.

At the very least nodes should refuse to accept new shards if they
can't fit them in the FS taking into account other allocated (but not yet
assigned) shards

On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann v.kiss...@gmail.comwrote:

Hi Otis,
i meant both with "excellent". Excellent for us, because our data was
still there. Not excellent for our AWS bill, we had an traffic over 100TB
in a couple of days.
The shards were really bounced around the cluster because every node
was running out of disk space. So ES tried to relocate the shards all the
time.
Cheers,
Vadim

Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic:

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network (rebalancing)"?
It almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**inde
**x.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-a
nalytics/index.html http://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote:

Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/e
lasticsearch-equilibriumhttps://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented
in ElasticSearch itself?

In particular, ES will try and allocate shards on machines which
are low on disk. We are seeing this all the time and I have to relocate
shards manually to prevent that. Why isn't there code in place to prevent
that?

Itamar.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Ivan, I hear you, but at the very least some free space calculation needs
to be in effect. For example, when ES hits a lower limit (say, 500MB of
free space) it should start relocation out or cancel relocations going in.

We are using the rolling indices technique, and our largest indexes are of
about 30GB in size (times the number of replicas for each), having 1 shard
per index. The shards per node won't really help us.

We also notice indices of such sizes take a lot of time to be initialized
once the node has been restarted. I wonder if there's any way of speeding
this up?

On Sat, Jul 27, 2013 at 12:18 AM, ppearcy ppearcy@gmail.com wrote:

I think you need to ensure that larger indexes have more shards, so the
data can be more easily distributed.

If you are running 0.90+ it should automatically ensure that shards for an
index are evenly (for the most part distributed):
https://github.com/elasticsearch/elasticsearch/issues/2555

If that doesn't work for you, you can manually force max N shards per node
with this setting:
index.routing.allocation.total_shards_per_node

Best Regards,
Paul

On Friday, July 26, 2013 10:44:16 AM UTC-6, Ivan Brusic wrote:

Merging in Lucene can take up a considerable amount of disk space for
intermediate segments depending on the merge settings. It would be hard to
take disk size into consideration for allocation without thinking of the
merge policy as well. I can see this issue being exacerbated with
single-shard indices (which luckily I do not have).

--
Ivan

On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko ita...@code972.comwrote:

No, not sure what issue that would be. Also haven't heard back on this
thread or on other issues I've opened, so....

On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic <otis.gos...@gmail.com

wrote:

Hi Itamar,

Sounds scary... did you open an issue we could watch?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**
index.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html

On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote:

Unfortunately we are seeing very undesired behavior - I do hope
ElasticSearch developers will act quickly on this as this seems like a huge
show stopper

Yes, ElasticSearch will not lose data (at least no data loss that we
were able to observe), but the current shard allocation behavior has some
very undesirable effects

We have couple of dozen of nodes, and whenever we add more nodes the
cluster tries to rebalance. Whenever this occurs, it is very easy to notice
a very flawed balancing function is used, and it takes a couple of manual
shard allocation commands to stabilize the cluster again.

We have some 400+ indexes with one shard each and some with multiple
replicas, some very big (10gb at least in size), some medium (around 5gb)
and many very small ones. Apparently the sharding function tries to balance
the cluster based of number of indexes on a server, without taking into
account its size on disk or number of documents in it.

I has come to a point where we get to 0kb free on some of the nodes
during normal operation and ES unable to flush to disk. This is pretty much
unacceptable, considering we have nodes with 150gb+ free in the same
cluster.

At the very least nodes should refuse to accept new shards if they
can't fit them in the FS taking into account other allocated (but not yet
assigned) shards

On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann <v.kiss...@gmail.com

wrote:

Hi Otis,
i meant both with "excellent". Excellent for us, because our data
was still there. Not excellent for our AWS bill, we had an traffic over
100TB in a couple of days.
The shards were really bounced around the cluster because every node
was running out of disk space. So ES tried to relocate the shards all the
time.
Cheers,
Vadim

Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic:

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network (rebalancing)"?
It almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**
inde**x.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-a

nalytics/index.htmlhttp://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann
wrote:

Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar
Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/e**
lasticsearch-equilibriumhttps://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't implemented
in ElasticSearch itself?

In particular, ES will try and allocate shards on machines which
are low on disk. We are seeing this all the time and I have to relocate
shards manually to prevent that. Why isn't there code in place to prevent
that?

Itamar.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@**googlegroups.**com.

For more options, visit https://groups.google.com/groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey Itamar,

I think you are absolutely right that we need to add some notion of disk
size / capacity etc. Regarding your question why this has not yet been
implemented, well I can only say that we are planning to expand the shard
allocation algorithm in the future and I am pretty sure that disk size /
capacity will play a major role here. I can imagine some sort of disk space
allocation decider that can restrict a node from allocating any further
shards given the used / free disk space and / or move shards away given a
certain limit etc. We can also make allocation decision based on the size
of the shards or move shards around once they fill up and we see that
certain shards are much bigger than others. All of these problems we need
to address and I think we should go ahead and take the input of this thread
here and open an issue, would you wanna open one and we discuss it further
on the issue?

simon

On Sunday, August 4, 2013 4:33:34 PM UTC+2, Itamar Syn-Hershko wrote:

Ivan, I hear you, but at the very least some free space calculation needs
to be in effect. For example, when ES hits a lower limit (say, 500MB of
free space) it should start relocation out or cancel relocations going in.

We are using the rolling indices technique, and our largest indexes are of
about 30GB in size (times the number of replicas for each), having 1 shard
per index. The shards per node won't really help us.

We also notice indices of such sizes take a lot of time to be initialized
once the node has been restarted. I wonder if there's any way of speeding
this up?

On Sat, Jul 27, 2013 at 12:18 AM, ppearcy <ppe...@gmail.com <javascript:>>wrote:

I think you need to ensure that larger indexes have more shards, so the
data can be more easily distributed.

If you are running 0.90+ it should automatically ensure that shards for
an index are evenly (for the most part distributed):
https://github.com/elasticsearch/elasticsearch/issues/2555

If that doesn't work for you, you can manually force max N shards per
node with this setting:
index.routing.allocation.total_shards_per_node

Best Regards,
Paul

On Friday, July 26, 2013 10:44:16 AM UTC-6, Ivan Brusic wrote:

Merging in Lucene can take up a considerable amount of disk space for
intermediate segments depending on the merge settings. It would be hard to
take disk size into consideration for allocation without thinking of the
merge policy as well. I can see this issue being exacerbated with
single-shard indices (which luckily I do not have).

--
Ivan

On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko ita...@code972.comwrote:

No, not sure what issue that would be. Also haven't heard back on this
thread or on other issues I've opened, so....

On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic <
otis.gos...@gmail.com> wrote:

Hi Itamar,

Sounds scary... did you open an issue we could watch?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**
index.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html

On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote:

Unfortunately we are seeing very undesired behavior - I do hope
ElasticSearch developers will act quickly on this as this seems like a huge
show stopper

Yes, ElasticSearch will not lose data (at least no data loss that we
were able to observe), but the current shard allocation behavior has some
very undesirable effects

We have couple of dozen of nodes, and whenever we add more nodes the
cluster tries to rebalance. Whenever this occurs, it is very easy to notice
a very flawed balancing function is used, and it takes a couple of manual
shard allocation commands to stabilize the cluster again.

We have some 400+ indexes with one shard each and some with multiple
replicas, some very big (10gb at least in size), some medium (around 5gb)
and many very small ones. Apparently the sharding function tries to balance
the cluster based of number of indexes on a server, without taking into
account its size on disk or number of documents in it.

I has come to a point where we get to 0kb free on some of the nodes
during normal operation and ES unable to flush to disk. This is pretty much
unacceptable, considering we have nodes with 150gb+ free in the same
cluster.

At the very least nodes should refuse to accept new shards if they
can't fit them in the FS taking into account other allocated (but not yet
assigned) shards

On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann <
v.kiss...@gmail.com> wrote:

Hi Otis,
i meant both with "excellent". Excellent for us, because our data
was still there. Not excellent for our AWS bill, we had an traffic over
100TB in a couple of days.
The shards were really bounced around the cluster because every node
was running out of disk space. So ES tried to relocate the shards all the
time.
Cheers,
Vadim

Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic:

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network (rebalancing)"?
It almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**
inde**x.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-a

nalytics/index.htmlhttp://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann
wrote:

Hi,
because ES relocate shards when your disk is full automatically.
We monitored an "excellent" behavior of ES last time, than all our AWS
nodes are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar
Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/e**
lasticsearch-equilibriumhttps://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't
implemented in ElasticSearch itself?

In particular, ES will try and allocate shards on machines which
are low on disk. We are seeing this all the time and I have to relocate
shards manually to prevent that. Why isn't there code in place to prevent
that?

Itamar.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@**googlegroups.**com.

For more options, visit https://groups.google.com/groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

On Tue, Aug 6, 2013 at 12:58 AM, simonw
simon.willnauer@elasticsearch.comwrote:

Hey Itamar,

I think you are absolutely right that we need to add some notion of disk
size / capacity etc. Regarding your question why this has not yet been
implemented, well I can only say that we are planning to expand the shard
allocation algorithm in the future and I am pretty sure that disk size /
capacity will play a major role here. I can imagine some sort of disk space
allocation decider that can restrict a node from allocating any further
shards given the used / free disk space and / or move shards away given a
certain limit etc. We can also make allocation decision based on the size
of the shards or move shards around once they fill up and we see that
certain shards are much bigger than others. All of these problems we need
to address and I think we should go ahead and take the input of this thread
here and open an issue, would you wanna open one and we discuss it further
on the issue?

simon

On Sunday, August 4, 2013 4:33:34 PM UTC+2, Itamar Syn-Hershko wrote:

Ivan, I hear you, but at the very least some free space calculation needs
to be in effect. For example, when ES hits a lower limit (say, 500MB of
free space) it should start relocation out or cancel relocations going in.

We are using the rolling indices technique, and our largest indexes are
of about 30GB in size (times the number of replicas for each), having 1
shard per index. The shards per node won't really help us.

We also notice indices of such sizes take a lot of time to be initialized
once the node has been restarted. I wonder if there's any way of speeding
this up?

On Sat, Jul 27, 2013 at 12:18 AM, ppearcy ppe...@gmail.com wrote:

I think you need to ensure that larger indexes have more shards, so the
data can be more easily distributed.

If you are running 0.90+ it should automatically ensure that shards for
an index are evenly (for the most part distributed):
https://github.com/**elasticsearch/elasticsearch/**issues/2555https://github.com/elasticsearch/elasticsearch/issues/2555

If that doesn't work for you, you can manually force max N shards per
node with this setting:
index.routing.allocation.**total_shards_per_node

Best Regards,
Paul

On Friday, July 26, 2013 10:44:16 AM UTC-6, Ivan Brusic wrote:

Merging in Lucene can take up a considerable amount of disk space for
intermediate segments depending on the merge settings. It would be hard to
take disk size into consideration for allocation without thinking of the
merge policy as well. I can see this issue being exacerbated with
single-shard indices (which luckily I do not have).

--
Ivan

On Thu, Jul 25, 2013 at 4:23 AM, Itamar Syn-Hershko <ita...@code972.com

wrote:

No, not sure what issue that would be. Also haven't heard back on this
thread or on other issues I've opened, so....

On Thu, Jul 25, 2013 at 1:49 PM, Otis Gospodnetic <
otis.gos...@gmail.com> wrote:

Hi Itamar,

Sounds scary... did you open an issue we could watch?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**inde
**x.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-a
nalytics/index.html http://sematext.com/search-analytics/index.html

On Sunday, July 21, 2013 1:11:15 PM UTC+2, Itamar Syn-Hershko wrote:

Unfortunately we are seeing very undesired behavior - I do hope
ElasticSearch developers will act quickly on this as this seems like a huge
show stopper

Yes, ElasticSearch will not lose data (at least no data loss that we
were able to observe), but the current shard allocation behavior has some
very undesirable effects

We have couple of dozen of nodes, and whenever we add more nodes the
cluster tries to rebalance. Whenever this occurs, it is very easy to notice
a very flawed balancing function is used, and it takes a couple of manual
shard allocation commands to stabilize the cluster again.

We have some 400+ indexes with one shard each and some with multiple
replicas, some very big (10gb at least in size), some medium (around 5gb)
and many very small ones. Apparently the sharding function tries to balance
the cluster based of number of indexes on a server, without taking into
account its size on disk or number of documents in it.

I has come to a point where we get to 0kb free on some of the nodes
during normal operation and ES unable to flush to disk. This is pretty much
unacceptable, considering we have nodes with 150gb+ free in the same
cluster.

At the very least nodes should refuse to accept new shards if they
can't fit them in the FS taking into account other allocated (but not yet
assigned) shards

On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann <
v.kiss...@gmail.com> wrote:

Hi Otis,
i meant both with "excellent". Excellent for us, because our data
was still there. Not excellent for our AWS bill, we had an traffic over
100TB in a couple of days.
The shards were really bounced around the cluster because every
node was running out of disk space. So ES tried to relocate the shards all
the time.
Cheers,
Vadim

Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis
Gospodnetic:

Hi Vadim,

I can't tell whether "excellent" is meant as emphasis or sarcasm?

What do you mean by "full index was help in network
(rebalancing)"? It almost sounds like you are saying shards were being
bounced around the cluster, but maybe you meant something else?

Otis

ELASTICSEARCH Performance Monitoring - http://sematext.com/spm/**
inde******x.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-a****
nalytics/index.htmlhttp://sematext.com/search-analytics/index.html

On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann
wrote:

Hi,
because ES relocate shards when your disk is full automatically.
We monitored an "excellent" behavior of ES last time, than all our AWS
nodes are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim

Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar
Syn-Hershko:

Hi,

I'm aware of https://github.com/sonian/e****
lasticsearch-equilibriumhttps://github.com/sonian/elasticsearch-equilibrium

But is there a reason why similar functionality isn't
implemented in ElasticSearch itself?

In particular, ES will try and allocate shards on machines which
are low on disk. We are seeing this all the time and I have to relocate
shards manually to prevent that. Why isn't there code in place to prevent
that?

Itamar.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@**googlegroups.com.

For more options, visit https://groups.google.com/grou**
ps/opt_out https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearc...@**googlegroups.**com.
For more options, visit https://groups.google.com/groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.**com.
For more options, visit https://groups.google.com/groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.