# Gateway recovery exception **URL:** \ **Category:** Elasticsearch\ **Created:** [March 14, 2012, 6:10pm UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007 "2012-03-14T18:10:34Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Johny\_John](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johny\_John](https://discuss.elastic.co/u/Johny_John)\ **Post date:** [March 14, 2012, 6:10pm UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007/1 "2012-03-14T18:10:34Z")
Hello, I manually deleted an index directory instead of playing a curl - XDELETE localhost:9200/indexName and i'm know stuck with a gateway recovery exception when i whant to restart the server : failed to start shard : org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [indexname][2] shard allocated for local recovery (post api), should exists, but doesn't The recevory precedure is looking for a shard that belongs to the deleted index. When i restart the server it recreates the index directory and the sub translog dir but everything is empty. What is the translog file to clean to prevent the recovery ? Is there any other solution ? Thank you for. Johny ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [March 15, 2012, 8:51am UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007/2 "2012-03-15T08:51:49Z")
Hi, It's unclear for me: is it possible to do a curl -XDELETE now? If so, that should solve it. If not, what directory did you remove? And what version of ES are you using? And how many nodes are in your cluster, with how many shards and replicas? I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ twitter/ on a single-node, and it restarted with no error (and no index, of course). Best regards, Radu On Mar 14, 8:10 pm, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > Hello, > > I manually deleted an index directory instead of playing a curl - > XDELETE localhost:9200/indexName and i'm know stuck with a gateway > recovery exception when i whant to restart the server : > > failed to start shard : > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [indexname][2] shard allocated for local recovery (post api), should > exists, but doesn't > > The recevory precedure is looking for a shard that belongs to the > deleted index. > > When i restart the server it recreates the index directory and the sub > translog dir but everything is empty. > > What is the translog file to clean to prevent the recovery ? > Is there any other solution ? > > Thank you for. > > Johny ---
**Author:** ![Johny\_John](https://avatars.discourse-cdn.com/v4/letter/j/ecb155/32.png) [@Johny\_John](https://discuss.elastic.co/u/Johny_John)\ **Post date:** [March 15, 2012, 9:51am UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007/3 "2012-03-15T09:51:42Z")
Hello, Thank you for the answer and sorry for the lack of explanations. I deleted an index folder manually It's now impossible du play a curl -XDELETE because the server is in recovery mode and not allowing requests. All is running as elasticsearch user using elasticsearch 0.17.8 with the following configuration : cluster.name: elasticsearch path.data: /home/elasticsearch path.work: /home/elasticsearch/work path.logs: /var/log/elasticsearch gateway.recover\_after\_nodes: 1 gateway.recover\_after\_time: 5m gateway.expected\_nodes: 1 I'm playing on a test server with 1 node and the default configuration for indices (5 shards and 1 replica). According to the documentation "The gateway.expected\_nodes allows to set how many nodes are expected to be in the cluster" so i changed the gateway.expected\_nodes from 2 to 1. If i let the value of 2 for this configuration key, the server starts correctly because it's not trying to recover. I tried a : curl -XDELETE localhost:9200/myindex but i get the following response : {"error":"IndexMissingException[[myindex] missing]","status":404} It's logical as the index does not exist anymore. With the gateway.expected\_nodes at 1, i'm in recovery mode, flooding the log file : [WARN][indices.cluster] [Sunturion] [myindex[0] failed to start shard 13 org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: [myindex][0] shard allocated for local recovery (post api), should exists, but doesn't 14 at org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java: 99) 15 at org.elasticsearch.index.gateway.IndexShardGatewayService $1.run(IndexShardGatewayService.java:179) 16 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: 1110) 17 at java.util.concurrent.ThreadPoolExecutor $Worker.run(ThreadPoolExecutor.java:603) 18 at java.lang.Thread.run(Thread.java:636) [WARN][cluster.action.shard] [Sunturion] sending failed shard for [myindex][0], node[UAmkAQe\_TUusC\_OvLEtoxg], [P], s[INITIALIZING], reason [Failed to start shard , message [IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard allocated for local recovery (post api), should exists, but doesn't]]] These errors are repeated for every shards of the index. Is there an error in my way of using the gateway ? If not, how can i tell the gateway not to recover the deleted index ? Thank you. Johny On 15 mar, 09:51, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > Hi, > > It's unclear for me: is it possible to do a curl -XDELETE now? If so, > that should solve it. > > If not, what directory did you remove? And what version of ES are you > using? And how many nodes are in your cluster, with how many shards > and replicas? > > I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ > twitter/ on a single-node, and it restarted with no error (and no > index, of course). > > Best regards, > Radu > > On Mar 14, 8:10 pm, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > Hello, > > > I manually deleted an index directory instead of playing a curl - > > XDELETE localhost:9200/indexName and i'm know stuck with a gateway > > recovery exception when i whant to restart the server : > > > failed to start shard : > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > [indexname][2] shard allocated for local recovery (post api), should > > exists, but doesn't > > > The recevory precedure is looking for a shard that belongs to the > > deleted index. > > > When i restart the server it recreates the index directory and the sub > > translog dir but everything is empty. > > > What is the translog file to clean to prevent the recovery ? > > Is there any other solution ? > > > Thank you for. > > > Johny ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [March 16, 2012, 7:40am UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007/4 "2012-03-16T07:40:15Z")
Hi John, Unfortunately, I don't know the answer to your questions, but since you say you're running on a test machine, it might be worth upgrading to 0.19 and see if the behavior gets solved. I've just tried with 0.19 with the configuration options you wrote, and when I manually removed the index directory, ES recovered the other index upon restart. So maybe there was a bug that was fixed in the meantime. Best regards, Radu On Mar 15, 11:51 am, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > Hello, > > Thank you for the answer and sorry for the lack of explanations. > > I deleted an index folder manually > It's now impossible du play a curl -XDELETE because the server is in > recovery mode and not allowing requests. > > All is running as elasticsearch user using elasticsearch 0.17.8 with > the following configuration : > > cluster.name: elasticsearch > > path.data: /home/elasticsearch > path.work: /home/elasticsearch/work > path.logs: /var/log/elasticsearch > > gateway.recover\_after\_nodes: 1 > gateway.recover\_after\_time: 5m > gateway.expected\_nodes: 1 > > I'm playing on a test server with 1 node and the default configuration > for indices (5 shards and 1 replica). > According to the documentation "The gateway.expected\_nodes allows to > set how many nodes are expected to be in the cluster" so i changed the > gateway.expected\_nodes from 2 to 1. > If i let the value of 2 for this configuration key, the server starts > correctly because it's not trying to recover. > I tried a : > curl -XDELETE localhost:9200/myindex > but i get the following response : > {"error":"IndexMissingException[[myindex] missing]","status":404} > > It's logical as the index does not exist anymore. > > With the gateway.expected\_nodes at 1, i'm in recovery mode, flooding > the log file : > > [WARN][indices.cluster] [Sunturion] [myindex[0] failed to > start shard > 13 > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > [myindex][0] shard allocated for local recovery (post api), should > exists, but doesn't > 14 at > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java: > 99) > 15 at > org.elasticsearch.index.gateway.IndexShardGatewayService > $1.run(IndexShardGatewayService.java:179) > 16 at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1110) > 17 at java.util.concurrent.ThreadPoolExecutor > $Worker.run(ThreadPoolExecutor.java:603) > 18 at java.lang.Thread.run(Thread.java:636) > > [WARN][cluster.action.shard] [Sunturion] sending failed shard > for [myindex][0], node[UAmkAQe\_TUusC\_OvLEtoxg], [P], s[INITIALIZING], > reason [Failed to start shard , message > [IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard > allocated for local recovery (post api), should exists, but doesn't]]] > > These errors are repeated for every shards of the index. > > Is there an error in my way of using the gateway ? > If not, how can i tell the gateway not to recover the deleted index ? > > Thank you. > > Johny > > On 15 mar, 09:51, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > Hi, > > > It's unclear for me: is it possible to do a curl -XDELETE now? If so, > > that should solve it. > > > If not, what directory did you remove? And what version of ES are you > > using? And how many nodes are in your cluster, with how many shards > > and replicas? > > > I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ > > twitter/ on a single-node, and it restarted with no error (and no > > index, of course). > > > Best regards, > > Radu > > > On Mar 14, 8:10 pm, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > > Hello, > > > > I manually deleted an index directory instead of playing a curl - > > > XDELETE localhost:9200/indexName and i'm know stuck with a gateway > > > recovery exception when i whant to restart the server : > > > > failed to start shard : > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > [indexname][2] shard allocated for local recovery (post api), should > > > exists, but doesn't > > > > The recevory precedure is looking for a shard that belongs to the > > > deleted index. > > > > When i restart the server it recreates the index directory and the sub > > > translog dir but everything is empty. > > > > What is the translog file to clean to prevent the recovery ? > > > Is there any other solution ? > > > > Thank you for. > > > > Johny ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 17, 2012, 10:01am UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007/5 "2012-03-17T10:01:13Z")
First, with the mentioned problem, you can simply issue a delete request (curl -XDELETE) and it will stop trying to recovery it. In 0.19, with the move to the new local gateway storage structure, if you delete the index from the file system, it will simply not be recovered any more (because the metadata is stored per index now, and not in a global file). Radu, this is why you saw the mentioned behavior with your recreation with 0.19. On Fri, Mar 16, 2012 at 9:40 AM, Radu Gheorghe [radu0gheorghe@gmail.com](mailto:radu0gheorghe@gmail.com)wrote: > Hi John, > > Unfortunately, I don't know the answer to your questions, but since > you say you're running on a test machine, it might be worth upgrading > to 0.19 and see if the behavior gets solved. > > I've just tried with 0.19 with the configuration options you wrote, > and when I manually removed the index directory, ES recovered the > other index upon restart. So maybe there was a bug that was fixed in > the meantime. > > Best regards, > Radu > > On Mar 15, 11:51 am, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > Hello, > > > > Thank you for the answer and sorry for the lack of explanations. > > > > I deleted an index folder manually > > It's now impossible du play a curl -XDELETE because the server is in > > recovery mode and not allowing requests. > > > > All is running as elasticsearch user using elasticsearch 0.17.8 with > > the following configuration : > > > > cluster.name: elasticsearch > > > > path.data: /home/elasticsearch > > path.work: /home/elasticsearch/work > > path.logs: /var/log/elasticsearch > > > > gateway.recover\_after\_nodes: 1 > > gateway.recover\_after\_time: 5m > > gateway.expected\_nodes: 1 > > > > I'm playing on a test server with 1 node and the default configuration > > for indices (5 shards and 1 replica). > > According to the documentation "The gateway.expected\_nodes allows to > > set how many nodes are expected to be in the cluster" so i changed the > > gateway.expected\_nodes from 2 to 1. > > If i let the value of 2 for this configuration key, the server starts > > correctly because it's not trying to recover. > > I tried a : > > curl -XDELETE localhost:9200/myindex > > but i get the following response : > > {"error":"IndexMissingException[[myindex] missing]","status":404} > > > > It's logical as the index does not exist anymore. > > > > With the gateway.expected\_nodes at 1, i'm in recovery mode, flooding > > the log file : > > > > [WARN][indices.cluster] [Sunturion] [myindex[0] failed to > > start shard > > 13 > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > [myindex][0] shard allocated for local recovery (post api), should > > exists, but doesn't > > 14 at > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java: > > > 1. > > > > 15 at > > org.elasticsearch.index.gateway.IndexShardGatewayService > > $1.run(IndexShardGatewayService.java:179) > > 16 at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > > > 1. > > > > 17 at java.util.concurrent.ThreadPoolExecutor > > $Worker.run(ThreadPoolExecutor.java:603) > > 18 at java.lang.Thread.run(Thread.java:636) > > > > [WARN][cluster.action.shard] [Sunturion] sending failed shard > > for [myindex][0], node[UAmkAQe\_TUusC\_OvLEtoxg], [P], s[INITIALIZING], > > reason [Failed to start shard , message > > [IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard > > allocated for local recovery (post api), should exists, but doesn't]]] > > > > These errors are repeated for every shards of the index. > > > > Is there an error in my way of using the gateway ? > > If not, how can i tell the gateway not to recover the deleted index ? > > > > Thank you. > > > > Johny > > > > On 15 mar, 09:51, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > Hi, > > > > > It's unclear for me: is it possible to do a curl -XDELETE now? If so, > > > that should solve it. > > > > > If not, what directory did you remove? And what version of ES are you > > > using? And how many nodes are in your cluster, with how many shards > > > and replicas? > > > > > I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ > > > twitter/ on a single-node, and it restarted with no error (and no > > > index, of course). > > > > > Best regards, > > > Radu > > > > > On Mar 14, 8:10 pm, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > > > > Hello, > > > > > > I manually deleted an index directory instead of playing a curl - > > > > XDELETE localhost:9200/indexName and i'm know stuck with a gateway > > > > recovery exception when i whant to restart the server : > > > > > > failed to start shard : > > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > > [indexname][2] shard allocated for local recovery (post api), should > > > > exists, but doesn't > > > > > > The recevory precedure is looking for a shard that belongs to the > > > > deleted index. > > > > > > When i restart the server it recreates the index directory and the > > > > sub > > > > translog dir but everything is empty. > > > > > > What is the translog file to clean to prevent the recovery ? > > > > Is there any other solution ? > > > > > > Thank you for. > > > > > > Johny ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [March 18, 2012, 2:44pm UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007/6 "2012-03-18T14:44:19Z")
Thanks for clearing it up, Shay! However, it seems that 'curl -XDELETE' isn't possible in this case. Is there another solution? Because I certainly don't see one. On 17 mar., 12:01, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > First, with the mentioned problem, you can simply issue a delete request > (curl -XDELETE) and it will stop trying to recovery it. > > In 0.19, with the move to the new local gateway storage structure, if you > delete the index from the file system, it will simply not be recovered any > more (because the metadata is stored per index now, and not in a global > file). Radu, this is why you saw the mentioned behavior with your > recreation with 0.19. > > On Fri, Mar 16, 2012 at 9:40 AM, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com)wrote: > > > Hi John, > > > Unfortunately, I don't know the answer to your questions, but since > > you say you're running on a test machine, it might be worth upgrading > > to 0.19 and see if the behavior gets solved. > > > I've just tried with 0.19 with the configuration options you wrote, > > and when I manually removed the index directory, ES recovered the > > other index upon restart. So maybe there was a bug that was fixed in > > the meantime. > > > Best regards, > > Radu > > > On Mar 15, 11:51 am, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > > > Hello, > > > > Thank you for the answer and sorry for the lack of explanations. > > > > I deleted an index folder manually > > > It's now impossible du play a curl -XDELETE because the server is in > > > recovery mode and not allowing requests. > > > > All is running as elasticsearch user using elasticsearch 0.17.8 with > > > the following configuration : > > > > cluster.name: elasticsearch > > > > path.data: /home/elasticsearch > > > path.work: /home/elasticsearch/work > > > path.logs: /var/log/elasticsearch > > > > gateway.recover\_after\_nodes: 1 > > > gateway.recover\_after\_time: 5m > > > gateway.expected\_nodes: 1 > > > > I'm playing on a test server with 1 node and the default configuration > > > for indices (5 shards and 1 replica). > > > According to the documentation "The gateway.expected\_nodes allows to > > > set how many nodes are expected to be in the cluster" so i changed the > > > gateway.expected\_nodes from 2 to 1. > > > If i let the value of 2 for this configuration key, the server starts > > > correctly because it's not trying to recover. > > > I tried a : > > > curl -XDELETE localhost:9200/myindex > > > but i get the following response : > > > {"error":"IndexMissingException[[myindex] missing]","status":404} > > > > It's logical as the index does not exist anymore. > > > > With the gateway.expected\_nodes at 1, i'm in recovery mode, flooding > > > the log file : > > > > [WARN][indices.cluster] [Sunturion] [myindex[0] failed to > > > start shard > > > 13 > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > [myindex][0] shard allocated for local recovery (post api), should > > > exists, but doesn't > > > 14 at > > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI­ndexShardGateway.java: > > > > > 1. > > > > > > 15 at > > > org.elasticsearch.index.gateway.IndexShardGatewayService > > > $1.run(IndexShardGatewayService.java:179) > > > 16 at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > > > > > 1. > > > > > > 17 at java.util.concurrent.ThreadPoolExecutor > > > $Worker.run(ThreadPoolExecutor.java:603) > > > 18 at java.lang.Thread.run(Thread.java:636) > > > > [WARN][cluster.action.shard] [Sunturion] sending failed shard > > > for [myindex][0], node[UAmkAQe\_TUusC\_OvLEtoxg], [P], s[INITIALIZING], > > > reason [Failed to start shard , message > > > [IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard > > > allocated for local recovery (post api), should exists, but doesn't]]] > > > > These errors are repeated for every shards of the index. > > > > Is there an error in my way of using the gateway ? > > > If not, how can i tell the gateway not to recover the deleted index ? > > > > Thank you. > > > > Johny > > > > On 15 mar, 09:51, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > Hi, > > > > > It's unclear for me: is it possible to do a curl -XDELETE now? If so, > > > > that should solve it. > > > > > If not, what directory did you remove? And what version of ES are you > > > > using? And how many nodes are in your cluster, with how many shards > > > > and replicas? > > > > > I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ > > > > twitter/ on a single-node, and it restarted with no error (and no > > > > index, of course). > > > > > Best regards, > > > > Radu > > > > > On Mar 14, 8:10 pm, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > > > > Hello, > > > > > > I manually deleted an index directory instead of playing a curl - > > > > > XDELETE localhost:9200/indexName and i'm know stuck with a gateway > > > > > recovery exception when i whant to restart the server : > > > > > > failed to start shard : > > > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > > > [indexname][2] shard allocated for local recovery (post api), should > > > > > exists, but doesn't > > > > > > The recevory precedure is looking for a shard that belongs to the > > > > > deleted index. > > > > > > When i restart the server it recreates the index directory and the > > > > > sub > > > > > translog dir but everything is empty. > > > > > > What is the translog file to clean to prevent the recovery ? > > > > > Is there any other solution ? > > > > > > Thank you for. > > > > > > Johny ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [March 20, 2012, 9:56am UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007/7 "2012-03-20T09:56:19Z")
What do you mean its not possible? Issue it, it might take time for it to apply because of the heavy rerouting going on, but it will be applied. On Sun, Mar 18, 2012 at 4:44 PM, Radu Gheorghe [radu0gheorghe@gmail.com](mailto:radu0gheorghe@gmail.com)wrote: > Thanks for clearing it up, Shay! > > However, it seems that 'curl -XDELETE' isn't possible in this case. Is > there another solution? Because I certainly don't see one. > > On 17 mar., 12:01, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > First, with the mentioned problem, you can simply issue a delete request > > (curl -XDELETE) and it will stop trying to recovery it. > > > > In 0.19, with the move to the new local gateway storage structure, if you > > delete the index from the file system, it will simply not be recovered > > any > > more (because the metadata is stored per index now, and not in a global > > file). Radu, this is why you saw the mentioned behavior with your > > recreation with 0.19. > > > > On Fri, Mar 16, 2012 at 9:40 AM, Radu Gheorghe \<[radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) > > wrote: > > > > > Hi John, > > > > > Unfortunately, I don't know the answer to your questions, but since > > > you say you're running on a test machine, it might be worth upgrading > > > to 0.19 and see if the behavior gets solved. > > > > > I've just tried with 0.19 with the configuration options you wrote, > > > and when I manually removed the index directory, ES recovered the > > > other index upon restart. So maybe there was a bug that was fixed in > > > the meantime. > > > > > Best regards, > > > Radu > > > > > On Mar 15, 11:51 am, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > > > > > Hello, > > > > > > Thank you for the answer and sorry for the lack of explanations. > > > > > > I deleted an index folder manually > > > > It's now impossible du play a curl -XDELETE because the server is in > > > > recovery mode and not allowing requests. > > > > > > All is running as elasticsearch user using elasticsearch 0.17.8 with > > > > the following configuration : > > > > > > cluster.name: elasticsearch > > > > > > path.data: /home/elasticsearch > > > > path.work: /home/elasticsearch/work > > > > path.logs: /var/log/elasticsearch > > > > > > gateway.recover\_after\_nodes: 1 > > > > gateway.recover\_after\_time: 5m > > > > gateway.expected\_nodes: 1 > > > > > > I'm playing on a test server with 1 node and the default > > > > configuration > > > > for indices (5 shards and 1 replica). > > > > According to the documentation "The gateway.expected\_nodes allows to > > > > set how many nodes are expected to be in the cluster" so i changed > > > > the > > > > gateway.expected\_nodes from 2 to 1. > > > > If i let the value of 2 for this configuration key, the server starts > > > > correctly because it's not trying to recover. > > > > I tried a : > > > > curl -XDELETE localhost:9200/myindex > > > > but i get the following response : > > > > {"error":"IndexMissingException[[myindex] missing]","status":404} > > > > > > It's logical as the index does not exist anymore. > > > > > > With the gateway.expected\_nodes at 1, i'm in recovery mode, flooding > > > > the log file : > > > > > > [WARN][indices.cluster] [Sunturion] [myindex[0] failed to > > > > start shard > > > > 13 > > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > > [myindex][0] shard allocated for local recovery (post api), should > > > > exists, but doesn't > > > > 14 at > > > > > > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI­ndexShardGateway.java: > > > > > 1. > > > > > > > > 15 at > > > > org.elasticsearch.index.gateway.IndexShardGatewayService > > > > $1.run(IndexShardGatewayService.java:179) > > > > 16 at > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > > > > > 1. > > > > > > > > 17 at java.util.concurrent.ThreadPoolExecutor > > > > $Worker.run(ThreadPoolExecutor.java:603) > > > > 18 at java.lang.Thread.run(Thread.java:636) > > > > > > [WARN][cluster.action.shard] [Sunturion] sending failed shard > > > > for [myindex][0], node[UAmkAQe\_TUusC\_OvLEtoxg], [P], s[INITIALIZING], > > > > reason [Failed to start shard , message > > > > [IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard > > > > allocated for local recovery (post api), should exists, but > > > > doesn't]]] > > > > > > These errors are repeated for every shards of the index. > > > > > > Is there an error in my way of using the gateway ? > > > > If not, how can i tell the gateway not to recover the deleted index ? > > > > > > Thank you. > > > > > > Johny > > > > > > On 15 mar, 09:51, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > > > Hi, > > > > > > > It's unclear for me: is it possible to do a curl -XDELETE now? If > > > > > so, > > > > > that should solve it. > > > > > > > If not, what directory did you remove? And what version of ES are > > > > > you > > > > > using? And how many nodes are in your cluster, with how many shards > > > > > and replicas? > > > > > > > I've just deleted > > > > > /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ > > > > > twitter/ on a single-node, and it restarted with no error (and no > > > > > index, of course). > > > > > > > Best regards, > > > > > Radu > > > > > > > On Mar 14, 8:10 pm, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > > > > > > Hello, > > > > > > > > I manually deleted an index directory instead of playing a curl - > > > > > > XDELETE localhost:9200/indexName and i'm know stuck with a > > > > > > gateway > > > > > > recovery exception when i whant to restart the server : > > > > > > > > failed to start shard : > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > > > > > [indexname][2] shard allocated for local recovery (post api), > > > > > > should > > > > > > exists, but doesn't > > > > > > > > The recevory precedure is looking for a shard that belongs to the > > > > > > deleted index. > > > > > > > > When i restart the server it recreates the index directory and > > > > > > the > > > > > > sub > > > > > > translog dir but everything is empty. > > > > > > > > What is the translog file to clean to prevent the recovery ? > > > > > > Is there any other solution ? > > > > > > > > Thank you for. > > > > > > > > Johny ---
**Author:** ![Radu\_Gheorghe1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe1/32/2688_2.png) [@Radu\_Gheorghe1](https://discuss.elastic.co/u/Radu_Gheorghe1)\ **Post date:** [March 24, 2012, 12:02pm UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007/8 "2012-03-24T12:02:17Z")
Oh, I see. Thanks for your reply! On 20 mar., 11:56, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > What do you mean its not possible? Issue it, it might take time for it to > apply because of the heavy rerouting going on, but it will be applied. > > On Sun, Mar 18, 2012 at 4:44 PM, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com)wrote: > > > Thanks for clearing it up, Shay! > > > However, it seems that 'curl -XDELETE' isn't possible in this case. Is > > there another solution? Because I certainly don't see one. > > > On 17 mar., 12:01, Shay Banon [kim...@gmail.com](mailto:kim...@gmail.com) wrote: > > > > > First, with the mentioned problem, you can simply issue a delete request > > > (curl -XDELETE) and it will stop trying to recovery it. > > > > In 0.19, with the move to the new local gateway storage structure, if you > > > delete the index from the file system, it will simply not be recovered > > > any > > > more (because the metadata is stored per index now, and not in a global > > > file). Radu, this is why you saw the mentioned behavior with your > > > recreation with 0.19. > > > > On Fri, Mar 16, 2012 at 9:40 AM, Radu Gheorghe \<[radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) > > > wrote: > > > > > Hi John, > > > > > Unfortunately, I don't know the answer to your questions, but since > > > > you say you're running on a test machine, it might be worth upgrading > > > > to 0.19 and see if the behavior gets solved. > > > > > I've just tried with 0.19 with the configuration options you wrote, > > > > and when I manually removed the index directory, ES recovered the > > > > other index upon restart. So maybe there was a bug that was fixed in > > > > the meantime. > > > > > Best regards, > > > > Radu > > > > > On Mar 15, 11:51 am, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > > > > > > > Hello, > > > > > > Thank you for the answer and sorry for the lack of explanations. > > > > > > I deleted an index folder manually > > > > > It's now impossible du play a curl -XDELETE because the server is in > > > > > recovery mode and not allowing requests. > > > > > > All is running as elasticsearch user using elasticsearch 0.17.8 with > > > > > the following configuration : > > > > > > cluster.name: elasticsearch > > > > > > path.data: /home/elasticsearch > > > > > path.work: /home/elasticsearch/work > > > > > path.logs: /var/log/elasticsearch > > > > > > gateway.recover\_after\_nodes: 1 > > > > > gateway.recover\_after\_time: 5m > > > > > gateway.expected\_nodes: 1 > > > > > > I'm playing on a test server with 1 node and the default > > > > > configuration > > > > > for indices (5 shards and 1 replica). > > > > > According to the documentation "The gateway.expected\_nodes allows to > > > > > set how many nodes are expected to be in the cluster" so i changed > > > > > the > > > > > gateway.expected\_nodes from 2 to 1. > > > > > If i let the value of 2 for this configuration key, the server starts > > > > > correctly because it's not trying to recover. > > > > > I tried a : > > > > > curl -XDELETE localhost:9200/myindex > > > > > but i get the following response : > > > > > {"error":"IndexMissingException[[myindex] missing]","status":404} > > > > > > It's logical as the index does not exist anymore. > > > > > > With the gateway.expected\_nodes at 1, i'm in recovery mode, flooding > > > > > the log file : > > > > > > [WARN][indices.cluster] [Sunturion] [myindex[0] failed to > > > > > start shard > > > > > 13 > > > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > > > [myindex][0] shard allocated for local recovery (post api), should > > > > > exists, but doesn't > > > > > 14 at > > > org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI­ndexShardGateway.java: > > > > > > > 1. > > > > > > > > > > 15 at > > > > > org.elasticsearch.index.gateway.IndexShardGatewayService > > > > > $1.run(IndexShardGatewayService.java:179) > > > > > 16 at > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > > > > > > > 1. > > > > > > > > > > 17 at java.util.concurrent.ThreadPoolExecutor > > > > > $Worker.run(ThreadPoolExecutor.java:603) > > > > > 18 at java.lang.Thread.run(Thread.java:636) > > > > > > [WARN][cluster.action.shard] [Sunturion] sending failed shard > > > > > for [myindex][0], node[UAmkAQe\_TUusC\_OvLEtoxg], [P], s[INITIALIZING], > > > > > reason [Failed to start shard , message > > > > > [IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard > > > > > allocated for local recovery (post api), should exists, but > > > > > doesn't]]] > > > > > > These errors are repeated for every shards of the index. > > > > > > Is there an error in my way of using the gateway ? > > > > > If not, how can i tell the gateway not to recover the deleted index ? > > > > > > Thank you. > > > > > > Johny > > > > > > On 15 mar, 09:51, Radu Gheorghe [radu0gheor...@gmail.com](mailto:radu0gheor...@gmail.com) wrote: > > > > > > > Hi, > > > > > > > It's unclear for me: is it possible to do a curl -XDELETE now? If > > > > > > so, > > > > > > that should solve it. > > > > > > > If not, what directory did you remove? And what version of ES are > > > > > > you > > > > > > using? And how many nodes are in your cluster, with how many shards > > > > > > and replicas? > > > > > > > I've just deleted > > > > > > /var/lib/elasticsearch/elasticsearch/nodes/0/indices/ > > > > > > twitter/ on a single-node, and it restarted with no error (and no > > > > > > index, of course). > > > > > > > Best regards, > > > > > > Radu > > > > > > > On Mar 14, 8:10 pm, Johny John [johnyjo...@gmail.com](mailto:johnyjo...@gmail.com) wrote: > > > > > > > > Hello, > > > > > > > > I manually deleted an index directory instead of playing a curl - > > > > > > > XDELETE localhost:9200/indexName and i'm know stuck with a > > > > > > > gateway > > > > > > > recovery exception when i whant to restart the server : > > > > > > > > failed to start shard : > > > org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException: > > > > > > > > > [indexname][2] shard allocated for local recovery (post api), > > > > > > > should > > > > > > > exists, but doesn't > > > > > > > > The recevory precedure is looking for a shard that belongs to the > > > > > > > deleted index. > > > > > > > > When i restart the server it recreates the index directory and > > > > > > > the > > > > > > > sub > > > > > > > translog dir but everything is empty. > > > > > > > > What is the translog file to clean to prevent the recovery ? > > > > > > > Is there any other solution ? > > > > > > > > Thank you for. > > > > > > > > Johny ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:35am UTC](https://discuss.elastic.co/t/gateway-recovery-exception/7007/9 "2017-07-06T03:35:02Z")