We encounter a lot of problems with cluster performance when trying to
scale it (adding a node to the cluster and also when removing one).
When you add a node the cluster starts to rebalance. The new node, once
shards are allocated, starts serving requests. Since we have a cluster with
constant indexing and queries, this causes an IO bottleneck on the new
node. We're on EC2, not using provisioned IOps - I know that can help but
wondering if there is some cluster configuration that might ease the load
on cluster scaling.
One thing I'm trying is reducing the cluster_concurrent_rebalance setting
to 1 to reduce moving shards.
There's also the throttling setting but I'm afraid that might impact
overall performance.
Any other recommendations on how to handle cluster scaling on a live
cluster?
The ideal situation was to be able to tell the cluster to copy shards to
the new node without transferring traffic and removing old shards yet, and
make the new node active only when rebalancing has finished. I guess that's
not a possibility right now...
I would recommend you to throttle recovery IO during adding a node to the
cluster with something like
"indices.recovery.max_size_per_sec" : 20mb
it might also make sense to set
"indices.recovery.concurrent_streams" : 1
simon
On Monday, January 21, 2013 7:42:10 PM UTC+1, Rotem wrote:
We encounter a lot of problems with cluster performance when trying to
scale it (adding a node to the cluster and also when removing one).
When you add a node the cluster starts to rebalance. The new node, once
shards are allocated, starts serving requests. Since we have a cluster with
constant indexing and queries, this causes an IO bottleneck on the new
node. We're on EC2, not using provisioned IOps - I know that can help but
wondering if there is some cluster configuration that might ease the load
on cluster scaling.
One thing I'm trying is reducing the cluster_concurrent_rebalance setting
to 1 to reduce moving shards.
There's also the throttling setting but I'm afraid that might impact
overall performance.
Any other recommendations on how to handle cluster scaling on a live
cluster?
The ideal situation was to be able to tell the cluster to copy shards to
the new node without transferring traffic and removing old shards yet, and
make the new node active only when rebalancing has finished. I guess that's
not a possibility right now...
I would recommend you to throttle recovery IO during adding a node to the
cluster with something like
"indices.recovery.max_size_**per_sec" : 20mb
it might also make sense to set
"indices.recovery.concurrent_**streams" : 1
simon
On Monday, January 21, 2013 7:42:10 PM UTC+1, Rotem wrote:
We encounter a lot of problems with cluster performance when trying to
scale it (adding a node to the cluster and also when removing one).
When you add a node the cluster starts to rebalance. The new node, once
shards are allocated, starts serving requests. Since we have a cluster with
constant indexing and queries, this causes an IO bottleneck on the new
node. We're on EC2, not using provisioned IOps - I know that can help but
wondering if there is some cluster configuration that might ease the load
on cluster scaling.
One thing I'm trying is reducing the cluster_concurrent_rebalancesetting to 1 to reduce moving shards.
There's also the throttling setting but I'm afraid that might impact
overall performance.
Any other recommendations on how to handle cluster scaling on a live
cluster?
The ideal situation was to be able to tell the cluster to copy shards to
the new node without transferring traffic and removing old shards yet, and
make the new node active only when rebalancing has finished. I guess that's
not a possibility right now...
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.