Unfortunately we are seeing very undesired behavior - I do hope
Elasticsearch developers will act quickly on this as this seems like a huge
show stopper
Yes, Elasticsearch will not lose data (at least no data loss that we were
able to observe), but the current shard allocation behavior has some very
undesirable effects
We have couple of dozen of nodes, and whenever we add more nodes the
cluster tries to rebalance. Whenever this occurs, it is very easy to notice
a very flawed balancing function is used, and it takes a couple of manual
shard allocation commands to stabilize the cluster again.
We have some 400+ indexes with one shard each and some with multiple
replicas, some very big (10gb at least in size), some medium (around 5gb)
and many very small ones. Apparently the sharding function tries to balance
the cluster based of number of indexes on a server, without taking into
account its size on disk or number of documents in it.
I has come to a point where we get to 0kb free on some of the nodes during
normal operation and ES unable to flush to disk. This is pretty much
unacceptable, considering we have nodes with 150gb+ free in the same
cluster.
At the very least nodes should refuse to accept new shards if they can't
fit them in the FS taking into account other allocated (but not yet
assigned) shards
On Thu, Jul 11, 2013 at 9:55 AM, Vadim Kisselmann v.kisselmann@gmail.comwrote:
Hi Otis,
i meant both with "excellent". Excellent for us, because our data was
still there. Not excellent for our AWS bill, we had an traffic over 100TB
in a couple of days.
The shards were really bounced around the cluster because every node was
running out of disk space. So ES tried to relocate the shards all the time.
Cheers,
Vadim
Am Donnerstag, 11. Juli 2013 04:23:10 UTC+2 schrieb Otis Gospodnetic:
Hi Vadim,
I can't tell whether "excellent" is meant as emphasis or sarcasm?
What do you mean by "full index was help in network (rebalancing)"? It
almost sounds like you are saying shards were being bounced around the
cluster, but maybe you meant something else?
Otis
ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service**
index.html http://sematext.com/spm/index.html
Search Analytics - http://sematext.com/search-**analytics/index.htmlhttp://sematext.com/search-analytics/index.html
On Wednesday, July 10, 2013 10:30:30 AM UTC-4, Vadim Kisselmann wrote:
Hi,
because ES relocate shards when your disk is full automatically. We
monitored an "excellent" behavior of ES last time, than all our AWS nodes
are running out of disk space. The full index was held in
network(rebalancing) till a new node was joined the cluster.
Cheers,
Vadim
Am Mittwoch, 10. Juli 2013 15:43:20 UTC+2 schrieb Itamar Syn-Hershko:
Hi,
I'm aware of https://github.com/sonian/**elasticsearch-equilibriumhttps://github.com/sonian/elasticsearch-equilibrium
But is there a reason why similar functionality isn't implemented in
Elasticsearch itself?
In particular, ES will try and allocate shards on machines which are
low on disk. We are seeing this all the time and I have to relocate shards
manually to prevent that. Why isn't there code in place to prevent that?
Itamar.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.