When re-opening a closed index, the cluster will move around the shards
even though the allocation should be even.
Most recent example: re-opening a 4-shard/1-replica index on an 8 node
cluster. Each node had before closing the index exactly 1 shard. After
re-opening, 4 of the shards remained where they were, but the other 4 moved
to recovering on different nodes. The primary delegation for some shards
also changed after re-opening, so it is not just that the replicas are
recovering.
Moving around just 1 replica of an index is a lot of network traffic. Is
there any way to avoid this situation?
Does each node still have one shard after the fact, they're just jumping
between nodes?
On Tuesday, December 4, 2012 11:03:38 AM UTC-8, Ivan Brusic wrote:
When re-opening a closed index, the cluster will move around the shards
even though the allocation should be even.
Most recent example: re-opening a 4-shard/1-replica index on an 8 node
cluster. Each node had before closing the index exactly 1 shard. After
re-opening, 4 of the shards remained where they were, but the other 4 moved
to recovering on different nodes. The primary delegation for some shards
also changed after re-opening, so it is not just that the replicas are
recovering.
Moving around just 1 replica of an index is a lot of network traffic. Is
there any way to avoid this situation?
Exactly. Each node in the cluster has the same amount of shards between all
indices before the one index was closed, and after it was re-opened.
Updates happen in bulk, so the index that was closed should not have any
updates before it was closed (in the minutes, if not more). Hopefully that
means the transaction log/gateway is clean.
--
Ivan
On Tue, Dec 4, 2012 at 12:42 PM, Dan Lecocq dan@seomoz.org wrote:
Does each node still have one shard after the fact, they're just jumping
between nodes?
On Tuesday, December 4, 2012 11:03:38 AM UTC-8, Ivan Brusic wrote:
When re-opening a closed index, the cluster will move around the shards
even though the allocation should be even.
Most recent example: re-opening a 4-shard/1-replica index on an 8 node
cluster. Each node had before closing the index exactly 1 shard. After
re-opening, 4 of the shards remained where they were, but the other 4 moved
to recovering on different nodes. The primary delegation for some shards
also changed after re-opening, so it is not just that the replicas are
recovering.
Moving around just 1 replica of an index is a lot of network traffic. Is
there any way to avoid this situation?
Are you using any of the cluster.routing.allocation.awareness settings?
Though, I'd expect that to only affect how they're initially assigned :-/
On Tuesday, December 4, 2012 1:49:08 PM UTC-8, Ivan Brusic wrote:
Exactly. Each node in the cluster has the same amount of shards between
all indices before the one index was closed, and after it was re-opened.
Updates happen in bulk, so the index that was closed should not have any
updates before it was closed (in the minutes, if not more). Hopefully that
means the transaction log/gateway is clean.
--
Ivan
On Tue, Dec 4, 2012 at 12:42 PM, Dan Lecocq <d...@seomoz.org <javascript:>
wrote:
Does each node still have one shard after the fact, they're just jumping
between nodes?
On Tuesday, December 4, 2012 11:03:38 AM UTC-8, Ivan Brusic wrote:
When re-opening a closed index, the cluster will move around the shards
even though the allocation should be even.
Most recent example: re-opening a 4-shard/1-replica index on an 8 node
cluster. Each node had before closing the index exactly 1 shard. After
re-opening, 4 of the shards remained where they were, but the other 4 moved
to recovering on different nodes. The primary delegation for some shards
also changed after re-opening, so it is not just that the replicas are
recovering.
Moving around just 1 replica of an index is a lot of network traffic. Is
there any way to avoid this situation?
Did not change the awareness settings. I did notice that gateway.expected_nodes
is set to 6 (on a 8 node cluster). Not the culprit, but something I should
fix.
Is this issue with the allocator or recovery? Does closing an index
automatically flush the index?
On Tue, Dec 4, 2012 at 2:00 PM, Dan Lecocq dan@seomoz.org wrote:
Are you using any of the cluster.routing.allocation.awareness settings?
Though, I'd expect that to only affect how they're initially assigned :-/
On Tuesday, December 4, 2012 1:49:08 PM UTC-8, Ivan Brusic wrote:
Exactly. Each node in the cluster has the same amount of shards between
all indices before the one index was closed, and after it was re-opened.
Updates happen in bulk, so the index that was closed should not have any
updates before it was closed (in the minutes, if not more). Hopefully that
means the transaction log/gateway is clean.
--
Ivan
On Tue, Dec 4, 2012 at 12:42 PM, Dan Lecocq d...@seomoz.org wrote:
Does each node still have one shard after the fact, they're just jumping
between nodes?
On Tuesday, December 4, 2012 11:03:38 AM UTC-8, Ivan Brusic wrote:
When re-opening a closed index, the cluster will move around the shards
even though the allocation should be even.
Most recent example: re-opening a 4-shard/1-replica index on an 8 node
cluster. Each node had before closing the index exactly 1 shard. After
re-opening, 4 of the shards remained where they were, but the other 4 moved
to recovering on different nodes. The primary delegation for some shards
also changed after re-opening, so it is not just that the replicas are
recovering.
Moving around just 1 replica of an index is a lot of network traffic.
Is there any way to avoid this situation?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.