Hi Shay,
I was optimistic about your hint, but apparently it isn't working in
the snapshot either.
As a bit of context, I configured local gateway, 1 shard, 0 replicas
in two different ES instances, E1 and E2, with independent work paths
ie: /work/es1 and /work/es2. The two instances were started with ./
elasticsearch -f, and were found and bound correctly.
I added 6 documents, 3 to index twitter1, 3 to index twitter2,
refreshed, flushed, and queried the servers to see everything was OK.
Then I killed both instances with "kill -9 PID1 PID2". I started only
ES1, which contained index twitter2.
Effectively index twitter1 was not found, some command line output
after recovering only ES1 with index twitter2:
$ curl -XGET 'http://sd:5100/_cluster/health?pretty=true'
{
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
}
$ curl -XGET 'http://sd:5100/_cluster/health/twitter1?pretty=true'
{
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 1,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1
}
$ curl -XGET 'http://sd:5100/_cluster/health/twitter2?pretty=true'
{
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0
}
So everything is probably as expected, twitter1 is in red state,
twitter2 is green, the whole cluster is in red state, but when I try
to delete twitter1 by the API:
$ curl -XDELETE 'http://sd:5100/twitter1/'
{"error":"ClusterBlockException[blocked by: [3/index not recovered
(not enough nodes with shards allocated found)];]"}
The message is OK, the index wasn't recovered, but I am telling ES to
delete it anyway, so I can recreate it. Maybe there is another way to
achieve this, am I missing something?
Thanks,
Sebastian
On Oct 14, 7:33 pm, Shay Banon shay.ba...@elasticsearch.com wrote:
Hi Sebastian,
I think you hit the bug in local gateway with several indices. What you
suggest is the behavior that should be. If there is a single shard missing,
not the whole cluster should be blocked, but only the index the shard is
missing for. In this case, you should be able to execute a single delete
index for that specific rouge index, and then reindex it. Can you test the
scenario with master (soon to be 0.12), I have tested it, just to see that
it meets what you are after.
-shay.banon
On Thu, Oct 14, 2010 at 10:15 PM, Sebastian sgavar...@gmail.com wrote:
Hi all,
I have a suggestion I'd like to talk about for a feature request.
When a shard can't be found on startup, the whole cluster is in
blocked state (I tested against v0.11), not only add operations, but
also the admin API. This prevents any type of programmable recovery,
or even hands on intervention using the APIs, and in fact the only
thing I found that works is to shutdown ES, delete all the indices
from the work path, start ES back, and rebuild them from scratch (all
this manually).
So, suggestion, could the cluster be left at least in read-only state
instead of blocked? and the admin API enabled, so an index delete is
still doable from the API?
As a side note, if there are many indices, with different degrees of
criticality, and only a single shard isn't found, you get all of them
blocked, so the way to minimize the possible blockings, with the
current options, would be to use different clusters on the same
network, which I don't think is optimal.
I am very happy with the functionality so far, but I think that
recovery isn't strong enough right now, and in a production system
things tend to go wrong from time to time.
Thanks,
Sebastian.