I've verified that shards are re-allocating after a cluster restart (again,
I'm using 1.0 RC1).
To test this specifically, I loaded a small dataset (can take a very long
time to verify results on a large dataset).
Easy to verify.
In a 5 node cluster, load some apache data. (I loaded only a couple dozen
days)
Let the cluster run until all shards are allocated, es-head can be good for
this.
Flush and shutdown the cluster.
Bring up only one node and point es-head at it, should display all 5 shards
for each index residing on the lone active node.
Bring up one additional, then maybe even a second node, refreshing es-head
every 15 seconds or so. Shards are observed first replicating to the second
node, then when the third node is active the shards are again re-allocated
for balancing.
So, either the entry I made into elasticsearch.yml to disable shard
allocation is incorrect, or there is likely a bug.(Or, I might
fundamentally misunderstand what disabling shard re-allocation is supposed
to do).
Maybe I'll re-test on a 0.90 cluster to see if it behaves differently...
Tony
On Friday, February 7, 2014 4:29:57 PM UTC-8, Tony Su wrote:
Hi Ivan,
Thx.
Yes, I have been doing a flush before every cluster shutdown now.
Running ES 1.0 RC1
I have been doing rolling restarts because I have been unable to start all
nodes nearly at once and get all nodes to join even after extending the
timeout as I described. But, as I'm theorizing I'm speculating that doing a
rolling restart is contributing to the shards being re-allocated because
nodes that contain shards for the index may not appear soon enough.
Maybe the entry I made in elasticsearch.yml exactly as I described isn't
correct? I derived it from an ES source that described sending the command
using curl but I thought better to enter directly in elasticsearch.yml
I'll take a look at your link, thx.
Tony
On Friday, February 7, 2014 3:23:24 PM UTC-8, Ivan Brusic wrote:
Shard allocation should never happen if disable_allocation is enabled.
Which version are you using? Are you doing a rolling restart or a full
cluster restart?
Two things that might help. First is to execute a flush before
restarting. I believe mismatched transaction states will label a shard as
incorrect during a restart. Also play around with the recovery settings
[1]. Try setting gateway.recover_after_nodes (disabled by default).
[1]
Elasticsearch Platform — Find real-time answers at scale | Elastic
Cheers,
Ivan
On Fri, Feb 7, 2014 at 3:11 PM, Tony Su tony...@gmail.com wrote:
At first, I noticed what some have called "shard thrashing," ie during
startup shards are re-allocated as nodes come online.
Have implemented the following by either creating a new setting or
modifying existing settings in elasticsearch.yml
- Disable allocation altogether
cluster.routing.allocation.disable_allocation: true
- Avoid split-brain in the current 5 node cluster
discovery.zen.minimum_master_nodes: 3
3 Increased Discovery timeout
discovery.zen.ping.timeout: 100s
Specific Objective:
When a cluster restarts, try to force re-use of how the shards were
allocated before shutdown.
Attempt:
- Tried to increase the discovery.zen.minimum_master_nodes to 5 in a 5
node cluster with the idea that if a node could refuse to become
operational until all 5 nodes in the cluster were recognized.
Result:
Unfortunately, despite making this setting equal to the total number of
nodes in the cluster, I observed shard re-allocation at 4 of the 5 nodes
without waiting for the fifth node to come online. And, this is with
allocation disabled.
Would like an opinion whether what I'm trying to accomplish is even
possible to
- As much as possible to force a restarted cluster to use existing
shards as already allocated
- Start all at once rather than rolling node starts which contributes to
shard re-allocation.
TIA,
Tony
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/aadbb803-f78e-4ddf-a718-69d4a2792f12%40googlegroups.com
.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/60e41b4b-dc7e-4768-83aa-b095e50b2749%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.