# Cluster read-only instead of blocked, and Admin API available on missing shard **URL:** \ **Category:** Elasticsearch\ **Created:** [October 14, 2010, 8:15pm UTC](https://discuss.elastic.co/t/cluster-read-only-instead-of-blocked-and-admin-api-available-on-missing-shard/3443 "2010-10-14T20:15:15Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Sebastian\_Gavarini](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@Sebastian\_Gavarini](https://discuss.elastic.co/u/Sebastian_Gavarini)\ **Post date:** [October 14, 2010, 8:15pm UTC](https://discuss.elastic.co/t/cluster-read-only-instead-of-blocked-and-admin-api-available-on-missing-shard/3443/1 "2010-10-14T20:15:15Z")
Hi all, I have a suggestion I'd like to talk about for a feature request. When a shard can't be found on startup, the whole cluster is in blocked state (I tested against v0.11), not only add operations, but also the admin API. This prevents any type of programmable recovery, or even hands on intervention using the APIs, and in fact the only thing I found that works is to shutdown ES, delete all the indices from the work path, start ES back, and rebuild them from scratch (all this manually). So, suggestion, could the cluster be left at least in read-only state instead of blocked? and the admin API enabled, so an index delete is still doable from the API? As a side note, if there are many indices, with different degrees of criticality, and only a single shard isn't found, you get all of them blocked, so the way to minimize the possible blockings, with the current options, would be to use different clusters on the same network, which I don't think is optimal. I am very happy with the functionality so far, but I think that recovery isn't strong enough right now, and in a production system things tend to go wrong from time to time. Thanks, Sebastian. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 14, 2010, 10:33pm UTC](https://discuss.elastic.co/t/cluster-read-only-instead-of-blocked-and-admin-api-available-on-missing-shard/3443/2 "2010-10-14T22:33:25Z")
Hi Sebastian, I think you hit the bug in local gateway with several indices. What you suggest is the behavior that should be. If there is a single shard missing, not the whole cluster should be blocked, but only the index the shard is missing for. In this case, you should be able to execute a single delete index for that specific rouge index, and then reindex it. Can you test the scenario with master (soon to be 0.12), I have tested it, just to see that it meets what you are after. -shay.banon On Thu, Oct 14, 2010 at 10:15 PM, Sebastian [sgavarini@gmail.com](mailto:sgavarini@gmail.com) wrote: > Hi all, > > I have a suggestion I'd like to talk about for a feature request. > > When a shard can't be found on startup, the whole cluster is in > blocked state (I tested against v0.11), not only add operations, but > also the admin API. This prevents any type of programmable recovery, > or even hands on intervention using the APIs, and in fact the only > thing I found that works is to shutdown ES, delete all the indices > from the work path, start ES back, and rebuild them from scratch (all > this manually). > > So, suggestion, could the cluster be left at least in read-only state > instead of blocked? and the admin API enabled, so an index delete is > still doable from the API? > > As a side note, if there are many indices, with different degrees of > criticality, and only a single shard isn't found, you get all of them > blocked, so the way to minimize the possible blockings, with the > current options, would be to use different clusters on the same > network, which I don't think is optimal. > > I am very happy with the functionality so far, but I think that > recovery isn't strong enough right now, and in a production system > things tend to go wrong from time to time. > > Thanks, > Sebastian. ---
**Author:** ![Sebastian\_Gavarini](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@Sebastian\_Gavarini](https://discuss.elastic.co/u/Sebastian_Gavarini)\ **Post date:** [October 15, 2010, 2:24am UTC](https://discuss.elastic.co/t/cluster-read-only-instead-of-blocked-and-admin-api-available-on-missing-shard/3443/3 "2010-10-15T02:24:15Z")
Hi Shay, I was optimistic about your hint, but apparently it isn't working in the snapshot either. As a bit of context, I configured local gateway, 1 shard, 0 replicas in two different ES instances, E1 and E2, with independent work paths ie: /work/es1 and /work/es2. The two instances were started with ./ elasticsearch -f, and were found and bound correctly. I added 6 documents, 3 to index twitter1, 3 to index twitter2, refreshed, flushed, and queried the servers to see everything was OK. Then I killed both instances with "kill -9 PID1 PID2". I started only ES1, which contained index twitter2. Effectively index twitter1 was not found, some command line output after recovering only ES1 with index twitter2: $ curl -XGET '[http://sd:5100/\_cluster/health?pretty=true](http://sd:5100/_cluster/health?pretty=true)' { "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 1, "active\_primary\_shards" : 1, "active\_shards" : 1, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 1 } $ curl -XGET '[http://sd:5100/\_cluster/health/twitter1?pretty=true](http://sd:5100/_cluster/health/twitter1?pretty=true)' { "status" : "red", "timed\_out" : false, "number\_of\_nodes" : 1, "active\_primary\_shards" : 0, "active\_shards" : 0, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 1 } $ curl -XGET '[http://sd:5100/\_cluster/health/twitter2?pretty=true](http://sd:5100/_cluster/health/twitter2?pretty=true)' { "status" : "green", "timed\_out" : false, "number\_of\_nodes" : 1, "active\_primary\_shards" : 1, "active\_shards" : 1, "relocating\_shards" : 0, "initializing\_shards" : 0, "unassigned\_shards" : 0 } So everything is probably as expected, twitter1 is in red state, twitter2 is green, the whole cluster is in red state, but when I try to delete twitter1 by the API: $ curl -XDELETE '[http://sd:5100/twitter1/](http://sd:5100/twitter1/)' {"error":"ClusterBlockException[blocked by: [3/index not recovered (not enough nodes with shards allocated found)];]"} The message is OK, the index wasn't recovered, but I am telling ES to delete it anyway, so I can recreate it. Maybe there is another way to achieve this, am I missing something? Thanks, Sebastian On Oct 14, 7:33 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Hi Sebastian, > > I think you hit the bug in local gateway with several indices. What you > suggest is the behavior that should be. If there is a single shard missing, > not the whole cluster should be blocked, but only the index the shard is > missing for. In this case, you should be able to execute a single delete > index for that specific rouge index, and then reindex it. Can you test the > scenario with master (soon to be 0.12), I have tested it, just to see that > it meets what you are after. > > -shay.banon > > On Thu, Oct 14, 2010 at 10:15 PM, Sebastian [sgavar...@gmail.com](mailto:sgavar...@gmail.com) wrote: > > > Hi all, > > > I have a suggestion I'd like to talk about for a feature request. > > > When a shard can't be found on startup, the whole cluster is in > > blocked state (I tested against v0.11), not only add operations, but > > also the admin API. This prevents any type of programmable recovery, > > or even hands on intervention using the APIs, and in fact the only > > thing I found that works is to shutdown ES, delete all the indices > > from the work path, start ES back, and rebuild them from scratch (all > > this manually). > > > So, suggestion, could the cluster be left at least in read-only state > > instead of blocked? and the admin API enabled, so an index delete is > > still doable from the API? > > > As a side note, if there are many indices, with different degrees of > > criticality, and only a single shard isn't found, you get all of them > > blocked, so the way to minimize the possible blockings, with the > > current options, would be to use different clusters on the same > > network, which I don't think is optimal. > > > I am very happy with the functionality so far, but I think that > > recovery isn't strong enough right now, and in a production system > > things tend to go wrong from time to time. > > > Thanks, > > Sebastian. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 15, 2010, 9:07am UTC](https://discuss.elastic.co/t/cluster-read-only-instead-of-blocked-and-admin-api-available-on-missing-shard/3443/4 "2010-10-15T09:07:31Z")
The optimism was in place, as all the problems you had were fixed except for the ability to delete an index that is not recovered ;). Pushed a fix for that: [Local Gateway: Don't block meta operations (delete index) on an index that is not recovered due to not all shards being available · Issue #433 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/433). -shay.banon On Fri, Oct 15, 2010 at 4:24 AM, Sebastian [sgavarini@gmail.com](mailto:sgavarini@gmail.com) wrote: > Hi Shay, > > I was optimistic about your hint, but apparently it isn't working in > the snapshot either. > > As a bit of context, I configured local gateway, 1 shard, 0 replicas > in two different ES instances, E1 and E2, with independent work paths > ie: /work/es1 and /work/es2. The two instances were started with ./ > elasticsearch -f, and were found and bound correctly. > > I added 6 documents, 3 to index twitter1, 3 to index twitter2, > refreshed, flushed, and queried the servers to see everything was OK. > Then I killed both instances with "kill -9 PID1 PID2". I started only > ES1, which contained index twitter2. > > Effectively index twitter1 was not found, some command line output > after recovering only ES1 with index twitter2: > > $ curl -XGET '[http://sd:5100/\_cluster/health?pretty=true](http://sd:5100/_cluster/health?pretty=true)' > { > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 1, > "active\_primary\_shards" : 1, > "active\_shards" : 1, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 1 > } > > $ curl -XGET '[http://sd:5100/\_cluster/health/twitter1?pretty=true](http://sd:5100/_cluster/health/twitter1?pretty=true)' > { > "status" : "red", > "timed\_out" : false, > "number\_of\_nodes" : 1, > "active\_primary\_shards" : 0, > "active\_shards" : 0, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 1 > } > > $ curl -XGET '[http://sd:5100/\_cluster/health/twitter2?pretty=true](http://sd:5100/_cluster/health/twitter2?pretty=true)' > { > "status" : "green", > "timed\_out" : false, > "number\_of\_nodes" : 1, > "active\_primary\_shards" : 1, > "active\_shards" : 1, > "relocating\_shards" : 0, > "initializing\_shards" : 0, > "unassigned\_shards" : 0 > } > > So everything is probably as expected, twitter1 is in red state, > twitter2 is green, the whole cluster is in red state, but when I try > to delete twitter1 by the API: > > $ curl -XDELETE '[http://sd:5100/twitter1/](http://sd:5100/twitter1/)' > {"error":"ClusterBlockException[blocked by: [3/index not recovered > (not enough nodes with shards allocated found)];]"} > > The message is OK, the index wasn't recovered, but I am telling ES to > delete it anyway, so I can recreate it. Maybe there is another way to > achieve this, am I missing something? > > Thanks, > Sebastian > > On Oct 14, 7:33 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Hi Sebastian, > > > > I think you hit the bug in local gateway with several indices. What you > > suggest is the behavior that should be. If there is a single shard > > missing, > > not the whole cluster should be blocked, but only the index the shard is > > missing for. In this case, you should be able to execute a single delete > > index for that specific rouge index, and then reindex it. Can you test > > the > > scenario with master (soon to be 0.12), I have tested it, just to see > > that > > it meets what you are after. > > > > -shay.banon > > > > On Thu, Oct 14, 2010 at 10:15 PM, Sebastian [sgavar...@gmail.com](mailto:sgavar...@gmail.com) wrote: > > > > > Hi all, > > > > > I have a suggestion I'd like to talk about for a feature request. > > > > > When a shard can't be found on startup, the whole cluster is in > > > blocked state (I tested against v0.11), not only add operations, but > > > also the admin API. This prevents any type of programmable recovery, > > > or even hands on intervention using the APIs, and in fact the only > > > thing I found that works is to shutdown ES, delete all the indices > > > from the work path, start ES back, and rebuild them from scratch (all > > > this manually). > > > > > So, suggestion, could the cluster be left at least in read-only state > > > instead of blocked? and the admin API enabled, so an index delete is > > > still doable from the API? > > > > > As a side note, if there are many indices, with different degrees of > > > criticality, and only a single shard isn't found, you get all of them > > > blocked, so the way to minimize the possible blockings, with the > > > current options, would be to use different clusters on the same > > > network, which I don't think is optimal. > > > > > I am very happy with the functionality so far, but I think that > > > recovery isn't strong enough right now, and in a production system > > > things tend to go wrong from time to time. > > > > > Thanks, > > > Sebastian. ---
**Author:** ![Sebastian\_Gavarini](https://avatars.discourse-cdn.com/v4/letter/s/db5fbb/32.png) [@Sebastian\_Gavarini](https://discuss.elastic.co/u/Sebastian_Gavarini)\ **Post date:** [October 15, 2010, 3:35pm UTC](https://discuss.elastic.co/t/cluster-read-only-instead-of-blocked-and-admin-api-available-on-missing-shard/3443/5 "2010-10-15T15:35:33Z")
Hi Shay, I've just run my test again, works like a charm now. Thanks a lot, Sebastian. On Oct 15, 6:07 am, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > The optimism was in place, as all the problems you had were fixed except for > the ability to delete an index that is not recovered ;). Pushed a fix for > that:[Local Gateway: Don't block meta operations (delete index) on an index that is not recovered due to not all shards being available · Issue #433 · elastic/elasticsearch · GitHub](http://github.com/elasticsearch/elasticsearch/issues/issue/433). > > -shay.banon > > On Fri, Oct 15, 2010 at 4:24 AM, Sebastian [sgavar...@gmail.com](mailto:sgavar...@gmail.com) wrote: > > > Hi Shay, > > > I was optimistic about your hint, but apparently it isn't working in > > the snapshot either. > > > As a bit of context, I configured local gateway, 1 shard, 0 replicas > > in two different ES instances, E1 and E2, with independent work paths > > ie: /work/es1 and /work/es2. The two instances were started with ./ > > elasticsearch -f, and were found and bound correctly. > > > I added 6 documents, 3 to index twitter1, 3 to index twitter2, > > refreshed, flushed, and queried the servers to see everything was OK. > > Then I killed both instances with "kill -9 PID1 PID2". I started only > > ES1, which contained index twitter2. > > > Effectively index twitter1 was not found, some command line output > > after recovering only ES1 with index twitter2: > > > $ curl -XGET '[http://sd:5100/\_cluster/health?pretty=true](http://sd:5100/_cluster/health?pretty=true)' > > { > > "status" : "red", > > "timed\_out" : false, > > "number\_of\_nodes" : 1, > > "active\_primary\_shards" : 1, > > "active\_shards" : 1, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 1 > > } > > > $ curl -XGET '[http://sd:5100/\_cluster/health/twitter1?pretty=true](http://sd:5100/_cluster/health/twitter1?pretty=true)' > > { > > "status" : "red", > > "timed\_out" : false, > > "number\_of\_nodes" : 1, > > "active\_primary\_shards" : 0, > > "active\_shards" : 0, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 1 > > } > > > $ curl -XGET '[http://sd:5100/\_cluster/health/twitter2?pretty=true](http://sd:5100/_cluster/health/twitter2?pretty=true)' > > { > > "status" : "green", > > "timed\_out" : false, > > "number\_of\_nodes" : 1, > > "active\_primary\_shards" : 1, > > "active\_shards" : 1, > > "relocating\_shards" : 0, > > "initializing\_shards" : 0, > > "unassigned\_shards" : 0 > > } > > > So everything is probably as expected, twitter1 is in red state, > > twitter2 is green, the whole cluster is in red state, but when I try > > to delete twitter1 by the API: > > > $ curl -XDELETE '[http://sd:5100/twitter1/](http://sd:5100/twitter1/)' > > {"error":"ClusterBlockException[blocked by: [3/index not recovered > > (not enough nodes with shards allocated found)];]"} > > > The message is OK, the index wasn't recovered, but I am telling ES to > > delete it anyway, so I can recreate it. Maybe there is another way to > > achieve this, am I missing something? > > > Thanks, > > Sebastian > > > On Oct 14, 7:33 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Hi Sebastian, > > > > I think you hit the bug in local gateway with several indices. What you > > > suggest is the behavior that should be. If there is a single shard > > > missing, > > > not the whole cluster should be blocked, but only the index the shard is > > > missing for. In this case, you should be able to execute a single delete > > > index for that specific rouge index, and then reindex it. Can you test > > > the > > > scenario with master (soon to be 0.12), I have tested it, just to see > > > that > > > it meets what you are after. > > > > -shay.banon > > > > On Thu, Oct 14, 2010 at 10:15 PM, Sebastian [sgavar...@gmail.com](mailto:sgavar...@gmail.com) wrote: > > > > > > > Hi all, > > > > > I have a suggestion I'd like to talk about for a feature request. > > > > > When a shard can't be found on startup, the whole cluster is in > > > > blocked state (I tested against v0.11), not only add operations, but > > > > also the admin API. This prevents any type of programmable recovery, > > > > or even hands on intervention using the APIs, and in fact the only > > > > thing I found that works is to shutdown ES, delete all the indices > > > > from the work path, start ES back, and rebuild them from scratch (all > > > > this manually). > > > > > So, suggestion, could the cluster be left at least in read-only state > > > > instead of blocked? and the admin API enabled, so an index delete is > > > > still doable from the API? > > > > > As a side note, if there are many indices, with different degrees of > > > > criticality, and only a single shard isn't found, you get all of them > > > > blocked, so the way to minimize the possible blockings, with the > > > > current options, would be to use different clusters on the same > > > > network, which I don't think is optimal. > > > > > I am very happy with the functionality so far, but I think that > > > > recovery isn't strong enough right now, and in a production system > > > > things tend to go wrong from time to time. > > > > > Thanks, > > > > Sebastian. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:18am UTC](https://discuss.elastic.co/t/cluster-read-only-instead-of-blocked-and-admin-api-available-on-missing-shard/3443/6 "2017-07-06T04:18:00Z")