Gateway recovery exception


(Johny John) #1

Hello,

I manually deleted an index directory instead of playing a curl -
XDELETE localhost:9200/indexName and i'm know stuck with a gateway
recovery exception when i whant to restart the server :

failed to start shard :
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[indexname][2] shard allocated for local recovery (post api), should
exists, but doesn't

The recevory precedure is looking for a shard that belongs to the
deleted index.

When i restart the server it recreates the index directory and the sub
translog dir but everything is empty.

What is the translog file to clean to prevent the recovery ?
Is there any other solution ?

Thank you for.

Johny


(Radu Gheorghe) #2

Hi,

It's unclear for me: is it possible to do a curl -XDELETE now? If so,
that should solve it.

If not, what directory did you remove? And what version of ES are you
using? And how many nodes are in your cluster, with how many shards
and replicas?

I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
twitter/ on a single-node, and it restarted with no error (and no
index, of course).

Best regards,
Radu

On Mar 14, 8:10 pm, Johny John johnyjo...@gmail.com wrote:

Hello,

I manually deleted an index directory instead of playing a curl -
XDELETE localhost:9200/indexName and i'm know stuck with a gateway
recovery exception when i whant to restart the server :

failed to start shard :
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[indexname][2] shard allocated for local recovery (post api), should
exists, but doesn't

The recevory precedure is looking for a shard that belongs to the
deleted index.

When i restart the server it recreates the index directory and the sub
translog dir but everything is empty.

What is the translog file to clean to prevent the recovery ?
Is there any other solution ?

Thank you for.

Johny


(Johny John) #3

Hello,

Thank you for the answer and sorry for the lack of explanations.

I deleted an index folder manually
It's now impossible du play a curl -XDELETE because the server is in
recovery mode and not allowing requests.

All is running as elasticsearch user using elasticsearch 0.17.8 with
the following configuration :

cluster.name: elasticsearch

path.data: /home/elasticsearch
path.work: /home/elasticsearch/work
path.logs: /var/log/elasticsearch

gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 1

I'm playing on a test server with 1 node and the default configuration
for indices (5 shards and 1 replica).
According to the documentation "The gateway.expected_nodes allows to
set how many nodes are expected to be in the cluster" so i changed the
gateway.expected_nodes from 2 to 1.
If i let the value of 2 for this configuration key, the server starts
correctly because it's not trying to recover.
I tried a :
curl -XDELETE localhost:9200/myindex
but i get the following response :
{"error":"IndexMissingException[[myindex] missing]","status":404}

It's logical as the index does not exist anymore.

With the gateway.expected_nodes at 1, i'm in recovery mode, flooding
the log file :

[WARN ][indices.cluster ] [Sunturion] [myindex[0] failed to
start shard
13
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[myindex][0] shard allocated for local recovery (post api), should
exists, but doesn't
14 at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:
99)
15 at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:179)
16 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
17 at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
18 at java.lang.Thread.run(Thread.java:636)

[WARN ][cluster.action.shard ] [Sunturion] sending failed shard
for [myindex][0], node[UAmkAQe_TUusC_OvLEtoxg], [P], s[INITIALIZING],
reason [Failed to start shard , message
[IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard
allocated for local recovery (post api), should exists, but doesn't]]]

These errors are repeated for every shards of the index.

Is there an error in my way of using the gateway ?
If not, how can i tell the gateway not to recover the deleted index ?

Thank you.

Johny

On 15 mar, 09:51, Radu Gheorghe radu0gheor...@gmail.com wrote:

Hi,

It's unclear for me: is it possible to do a curl -XDELETE now? If so,
that should solve it.

If not, what directory did you remove? And what version of ES are you
using? And how many nodes are in your cluster, with how many shards
and replicas?

I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
twitter/ on a single-node, and it restarted with no error (and no
index, of course).

Best regards,
Radu

On Mar 14, 8:10 pm, Johny John johnyjo...@gmail.com wrote:

Hello,

I manually deleted an index directory instead of playing a curl -
XDELETE localhost:9200/indexName and i'm know stuck with a gateway
recovery exception when i whant to restart the server :

failed to start shard :
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[indexname][2] shard allocated for local recovery (post api), should
exists, but doesn't

The recevory precedure is looking for a shard that belongs to the
deleted index.

When i restart the server it recreates the index directory and the sub
translog dir but everything is empty.

What is the translog file to clean to prevent the recovery ?
Is there any other solution ?

Thank you for.

Johny


(Radu Gheorghe) #4

Hi John,

Unfortunately, I don't know the answer to your questions, but since
you say you're running on a test machine, it might be worth upgrading
to 0.19 and see if the behavior gets solved.

I've just tried with 0.19 with the configuration options you wrote,
and when I manually removed the index directory, ES recovered the
other index upon restart. So maybe there was a bug that was fixed in
the meantime.

Best regards,
Radu

On Mar 15, 11:51 am, Johny John johnyjo...@gmail.com wrote:

Hello,

Thank you for the answer and sorry for the lack of explanations.

I deleted an index folder manually
It's now impossible du play a curl -XDELETE because the server is in
recovery mode and not allowing requests.

All is running as elasticsearch user using elasticsearch 0.17.8 with
the following configuration :

cluster.name: elasticsearch

path.data: /home/elasticsearch
path.work: /home/elasticsearch/work
path.logs: /var/log/elasticsearch

gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 1

I'm playing on a test server with 1 node and the default configuration
for indices (5 shards and 1 replica).
According to the documentation "The gateway.expected_nodes allows to
set how many nodes are expected to be in the cluster" so i changed the
gateway.expected_nodes from 2 to 1.
If i let the value of 2 for this configuration key, the server starts
correctly because it's not trying to recover.
I tried a :
curl -XDELETE localhost:9200/myindex
but i get the following response :
{"error":"IndexMissingException[[myindex] missing]","status":404}

It's logical as the index does not exist anymore.

With the gateway.expected_nodes at 1, i'm in recovery mode, flooding
the log file :

[WARN ][indices.cluster ] [Sunturion] [myindex[0] failed to
start shard
13
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[myindex][0] shard allocated for local recovery (post api), should
exists, but doesn't
14 at
org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:
99)
15 at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:179)
16 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:
1110)
17 at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
18 at java.lang.Thread.run(Thread.java:636)

[WARN ][cluster.action.shard ] [Sunturion] sending failed shard
for [myindex][0], node[UAmkAQe_TUusC_OvLEtoxg], [P], s[INITIALIZING],
reason [Failed to start shard , message
[IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard
allocated for local recovery (post api), should exists, but doesn't]]]

These errors are repeated for every shards of the index.

Is there an error in my way of using the gateway ?
If not, how can i tell the gateway not to recover the deleted index ?

Thank you.

Johny

On 15 mar, 09:51, Radu Gheorghe radu0gheor...@gmail.com wrote:

Hi,

It's unclear for me: is it possible to do a curl -XDELETE now? If so,
that should solve it.

If not, what directory did you remove? And what version of ES are you
using? And how many nodes are in your cluster, with how many shards
and replicas?

I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
twitter/ on a single-node, and it restarted with no error (and no
index, of course).

Best regards,
Radu

On Mar 14, 8:10 pm, Johny John johnyjo...@gmail.com wrote:

Hello,

I manually deleted an index directory instead of playing a curl -
XDELETE localhost:9200/indexName and i'm know stuck with a gateway
recovery exception when i whant to restart the server :

failed to start shard :
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[indexname][2] shard allocated for local recovery (post api), should
exists, but doesn't

The recevory precedure is looking for a shard that belongs to the
deleted index.

When i restart the server it recreates the index directory and the sub
translog dir but everything is empty.

What is the translog file to clean to prevent the recovery ?
Is there any other solution ?

Thank you for.

Johny


(Shay Banon) #5

First, with the mentioned problem, you can simply issue a delete request
(curl -XDELETE) and it will stop trying to recovery it.

In 0.19, with the move to the new local gateway storage structure, if you
delete the index from the file system, it will simply not be recovered any
more (because the metadata is stored per index now, and not in a global
file). Radu, this is why you saw the mentioned behavior with your
recreation with 0.19.

On Fri, Mar 16, 2012 at 9:40 AM, Radu Gheorghe radu0gheorghe@gmail.comwrote:

Hi John,

Unfortunately, I don't know the answer to your questions, but since
you say you're running on a test machine, it might be worth upgrading
to 0.19 and see if the behavior gets solved.

I've just tried with 0.19 with the configuration options you wrote,
and when I manually removed the index directory, ES recovered the
other index upon restart. So maybe there was a bug that was fixed in
the meantime.

Best regards,
Radu

On Mar 15, 11:51 am, Johny John johnyjo...@gmail.com wrote:

Hello,

Thank you for the answer and sorry for the lack of explanations.

I deleted an index folder manually
It's now impossible du play a curl -XDELETE because the server is in
recovery mode and not allowing requests.

All is running as elasticsearch user using elasticsearch 0.17.8 with
the following configuration :

cluster.name: elasticsearch

path.data: /home/elasticsearch
path.work: /home/elasticsearch/work
path.logs: /var/log/elasticsearch

gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 1

I'm playing on a test server with 1 node and the default configuration
for indices (5 shards and 1 replica).
According to the documentation "The gateway.expected_nodes allows to
set how many nodes are expected to be in the cluster" so i changed the
gateway.expected_nodes from 2 to 1.
If i let the value of 2 for this configuration key, the server starts
correctly because it's not trying to recover.
I tried a :
curl -XDELETE localhost:9200/myindex
but i get the following response :
{"error":"IndexMissingException[[myindex] missing]","status":404}

It's logical as the index does not exist anymore.

With the gateway.expected_nodes at 1, i'm in recovery mode, flooding
the log file :

[WARN ][indices.cluster ] [Sunturion] [myindex[0] failed to
start shard
13
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[myindex][0] shard allocated for local recovery (post api), should
exists, but doesn't
14 at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalIndexShardGateway.java:

15 at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:179)
16 at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

17 at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
18 at java.lang.Thread.run(Thread.java:636)

[WARN ][cluster.action.shard ] [Sunturion] sending failed shard
for [myindex][0], node[UAmkAQe_TUusC_OvLEtoxg], [P], s[INITIALIZING],
reason [Failed to start shard , message
[IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard
allocated for local recovery (post api), should exists, but doesn't]]]

These errors are repeated for every shards of the index.

Is there an error in my way of using the gateway ?
If not, how can i tell the gateway not to recover the deleted index ?

Thank you.

Johny

On 15 mar, 09:51, Radu Gheorghe radu0gheor...@gmail.com wrote:

Hi,

It's unclear for me: is it possible to do a curl -XDELETE now? If so,
that should solve it.

If not, what directory did you remove? And what version of ES are you
using? And how many nodes are in your cluster, with how many shards
and replicas?

I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
twitter/ on a single-node, and it restarted with no error (and no
index, of course).

Best regards,
Radu

On Mar 14, 8:10 pm, Johny John johnyjo...@gmail.com wrote:

Hello,

I manually deleted an index directory instead of playing a curl -
XDELETE localhost:9200/indexName and i'm know stuck with a gateway
recovery exception when i whant to restart the server :

failed to start shard :
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[indexname][2] shard allocated for local recovery (post api), should
exists, but doesn't

The recevory precedure is looking for a shard that belongs to the
deleted index.

When i restart the server it recreates the index directory and the
sub

translog dir but everything is empty.

What is the translog file to clean to prevent the recovery ?
Is there any other solution ?

Thank you for.

Johny


(Radu Gheorghe) #6

Thanks for clearing it up, Shay!

However, it seems that 'curl -XDELETE' isn't possible in this case. Is
there another solution? Because I certainly don't see one.

On 17 mar., 12:01, Shay Banon kim...@gmail.com wrote:

First, with the mentioned problem, you can simply issue a delete request
(curl -XDELETE) and it will stop trying to recovery it.

In 0.19, with the move to the new local gateway storage structure, if you
delete the index from the file system, it will simply not be recovered any
more (because the metadata is stored per index now, and not in a global
file). Radu, this is why you saw the mentioned behavior with your
recreation with 0.19.

On Fri, Mar 16, 2012 at 9:40 AM, Radu Gheorghe radu0gheor...@gmail.comwrote:

Hi John,

Unfortunately, I don't know the answer to your questions, but since
you say you're running on a test machine, it might be worth upgrading
to 0.19 and see if the behavior gets solved.

I've just tried with 0.19 with the configuration options you wrote,
and when I manually removed the index directory, ES recovered the
other index upon restart. So maybe there was a bug that was fixed in
the meantime.

Best regards,
Radu

On Mar 15, 11:51 am, Johny John johnyjo...@gmail.com wrote:

Hello,

Thank you for the answer and sorry for the lack of explanations.

I deleted an index folder manually
It's now impossible du play a curl -XDELETE because the server is in
recovery mode and not allowing requests.

All is running as elasticsearch user using elasticsearch 0.17.8 with
the following configuration :

cluster.name: elasticsearch

path.data: /home/elasticsearch
path.work: /home/elasticsearch/work
path.logs: /var/log/elasticsearch

gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 1

I'm playing on a test server with 1 node and the default configuration
for indices (5 shards and 1 replica).
According to the documentation "The gateway.expected_nodes allows to
set how many nodes are expected to be in the cluster" so i changed the
gateway.expected_nodes from 2 to 1.
If i let the value of 2 for this configuration key, the server starts
correctly because it's not trying to recover.
I tried a :
curl -XDELETE localhost:9200/myindex
but i get the following response :
{"error":"IndexMissingException[[myindex] missing]","status":404}

It's logical as the index does not exist anymore.

With the gateway.expected_nodes at 1, i'm in recovery mode, flooding
the log file :

[WARN ][indices.cluster ] [Sunturion] [myindex[0] failed to
start shard
13
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[myindex][0] shard allocated for local recovery (post api), should
exists, but doesn't
14 at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI­ndexShardGateway.java:

15 at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:179)
16 at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

17 at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
18 at java.lang.Thread.run(Thread.java:636)

[WARN ][cluster.action.shard ] [Sunturion] sending failed shard
for [myindex][0], node[UAmkAQe_TUusC_OvLEtoxg], [P], s[INITIALIZING],
reason [Failed to start shard , message
[IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard
allocated for local recovery (post api), should exists, but doesn't]]]

These errors are repeated for every shards of the index.

Is there an error in my way of using the gateway ?
If not, how can i tell the gateway not to recover the deleted index ?

Thank you.

Johny

On 15 mar, 09:51, Radu Gheorghe radu0gheor...@gmail.com wrote:

Hi,

It's unclear for me: is it possible to do a curl -XDELETE now? If so,
that should solve it.

If not, what directory did you remove? And what version of ES are you
using? And how many nodes are in your cluster, with how many shards
and replicas?

I've just deleted /var/lib/elasticsearch/elasticsearch/nodes/0/indices/
twitter/ on a single-node, and it restarted with no error (and no
index, of course).

Best regards,
Radu

On Mar 14, 8:10 pm, Johny John johnyjo...@gmail.com wrote:

Hello,

I manually deleted an index directory instead of playing a curl -
XDELETE localhost:9200/indexName and i'm know stuck with a gateway
recovery exception when i whant to restart the server :

failed to start shard :
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[indexname][2] shard allocated for local recovery (post api), should
exists, but doesn't

The recevory precedure is looking for a shard that belongs to the
deleted index.

When i restart the server it recreates the index directory and the
sub

translog dir but everything is empty.

What is the translog file to clean to prevent the recovery ?
Is there any other solution ?

Thank you for.

Johny


(Shay Banon) #7

What do you mean its not possible? Issue it, it might take time for it to
apply because of the heavy rerouting going on, but it will be applied.

On Sun, Mar 18, 2012 at 4:44 PM, Radu Gheorghe radu0gheorghe@gmail.comwrote:

Thanks for clearing it up, Shay!

However, it seems that 'curl -XDELETE' isn't possible in this case. Is
there another solution? Because I certainly don't see one.

On 17 mar., 12:01, Shay Banon kim...@gmail.com wrote:

First, with the mentioned problem, you can simply issue a delete request
(curl -XDELETE) and it will stop trying to recovery it.

In 0.19, with the move to the new local gateway storage structure, if you
delete the index from the file system, it will simply not be recovered
any
more (because the metadata is stored per index now, and not in a global
file). Radu, this is why you saw the mentioned behavior with your
recreation with 0.19.

On Fri, Mar 16, 2012 at 9:40 AM, Radu Gheorghe <radu0gheor...@gmail.com
wrote:

Hi John,

Unfortunately, I don't know the answer to your questions, but since
you say you're running on a test machine, it might be worth upgrading
to 0.19 and see if the behavior gets solved.

I've just tried with 0.19 with the configuration options you wrote,
and when I manually removed the index directory, ES recovered the
other index upon restart. So maybe there was a bug that was fixed in
the meantime.

Best regards,
Radu

On Mar 15, 11:51 am, Johny John johnyjo...@gmail.com wrote:

Hello,

Thank you for the answer and sorry for the lack of explanations.

I deleted an index folder manually
It's now impossible du play a curl -XDELETE because the server is in
recovery mode and not allowing requests.

All is running as elasticsearch user using elasticsearch 0.17.8 with
the following configuration :

cluster.name: elasticsearch

path.data: /home/elasticsearch
path.work: /home/elasticsearch/work
path.logs: /var/log/elasticsearch

gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 1

I'm playing on a test server with 1 node and the default
configuration

for indices (5 shards and 1 replica).
According to the documentation "The gateway.expected_nodes allows to
set how many nodes are expected to be in the cluster" so i changed
the

gateway.expected_nodes from 2 to 1.
If i let the value of 2 for this configuration key, the server starts
correctly because it's not trying to recover.
I tried a :
curl -XDELETE localhost:9200/myindex
but i get the following response :
{"error":"IndexMissingException[[myindex] missing]","status":404}

It's logical as the index does not exist anymore.

With the gateway.expected_nodes at 1, i'm in recovery mode, flooding
the log file :

[WARN ][indices.cluster ] [Sunturion] [myindex[0] failed to
start shard
13
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[myindex][0] shard allocated for local recovery (post api), should
exists, but doesn't
14 at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI­ndexShardGateway.java:

15 at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:179)
16 at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

17 at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
18 at java.lang.Thread.run(Thread.java:636)

[WARN ][cluster.action.shard ] [Sunturion] sending failed shard
for [myindex][0], node[UAmkAQe_TUusC_OvLEtoxg], [P], s[INITIALIZING],
reason [Failed to start shard , message
[IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard
allocated for local recovery (post api), should exists, but
doesn't]]]

These errors are repeated for every shards of the index.

Is there an error in my way of using the gateway ?
If not, how can i tell the gateway not to recover the deleted index ?

Thank you.

Johny

On 15 mar, 09:51, Radu Gheorghe radu0gheor...@gmail.com wrote:

Hi,

It's unclear for me: is it possible to do a curl -XDELETE now? If
so,

that should solve it.

If not, what directory did you remove? And what version of ES are
you

using? And how many nodes are in your cluster, with how many shards
and replicas?

I've just deleted
/var/lib/elasticsearch/elasticsearch/nodes/0/indices/

twitter/ on a single-node, and it restarted with no error (and no
index, of course).

Best regards,
Radu

On Mar 14, 8:10 pm, Johny John johnyjo...@gmail.com wrote:

Hello,

I manually deleted an index directory instead of playing a curl -
XDELETE localhost:9200/indexName and i'm know stuck with a
gateway

recovery exception when i whant to restart the server :

failed to start shard :

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[indexname][2] shard allocated for local recovery (post api),
should

exists, but doesn't

The recevory precedure is looking for a shard that belongs to the
deleted index.

When i restart the server it recreates the index directory and
the

sub

translog dir but everything is empty.

What is the translog file to clean to prevent the recovery ?
Is there any other solution ?

Thank you for.

Johny


(Radu Gheorghe) #8

Oh, I see. Thanks for your reply!

On 20 mar., 11:56, Shay Banon kim...@gmail.com wrote:

What do you mean its not possible? Issue it, it might take time for it to
apply because of the heavy rerouting going on, but it will be applied.

On Sun, Mar 18, 2012 at 4:44 PM, Radu Gheorghe radu0gheor...@gmail.comwrote:

Thanks for clearing it up, Shay!

However, it seems that 'curl -XDELETE' isn't possible in this case. Is
there another solution? Because I certainly don't see one.

On 17 mar., 12:01, Shay Banon kim...@gmail.com wrote:

First, with the mentioned problem, you can simply issue a delete request
(curl -XDELETE) and it will stop trying to recovery it.

In 0.19, with the move to the new local gateway storage structure, if you
delete the index from the file system, it will simply not be recovered
any
more (because the metadata is stored per index now, and not in a global
file). Radu, this is why you saw the mentioned behavior with your
recreation with 0.19.

On Fri, Mar 16, 2012 at 9:40 AM, Radu Gheorghe <radu0gheor...@gmail.com
wrote:

Hi John,

Unfortunately, I don't know the answer to your questions, but since
you say you're running on a test machine, it might be worth upgrading
to 0.19 and see if the behavior gets solved.

I've just tried with 0.19 with the configuration options you wrote,
and when I manually removed the index directory, ES recovered the
other index upon restart. So maybe there was a bug that was fixed in
the meantime.

Best regards,
Radu

On Mar 15, 11:51 am, Johny John johnyjo...@gmail.com wrote:

Hello,

Thank you for the answer and sorry for the lack of explanations.

I deleted an index folder manually
It's now impossible du play a curl -XDELETE because the server is in
recovery mode and not allowing requests.

All is running as elasticsearch user using elasticsearch 0.17.8 with
the following configuration :

cluster.name: elasticsearch

path.data: /home/elasticsearch
path.work: /home/elasticsearch/work
path.logs: /var/log/elasticsearch

gateway.recover_after_nodes: 1
gateway.recover_after_time: 5m
gateway.expected_nodes: 1

I'm playing on a test server with 1 node and the default
configuration

for indices (5 shards and 1 replica).
According to the documentation "The gateway.expected_nodes allows to
set how many nodes are expected to be in the cluster" so i changed
the

gateway.expected_nodes from 2 to 1.
If i let the value of 2 for this configuration key, the server starts
correctly because it's not trying to recover.
I tried a :
curl -XDELETE localhost:9200/myindex
but i get the following response :
{"error":"IndexMissingException[[myindex] missing]","status":404}

It's logical as the index does not exist anymore.

With the gateway.expected_nodes at 1, i'm in recovery mode, flooding
the log file :

[WARN ][indices.cluster ] [Sunturion] [myindex[0] failed to
start shard
13
org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:
[myindex][0] shard allocated for local recovery (post api), should
exists, but doesn't
14 at

org.elasticsearch.index.gateway.local.LocalIndexShardGateway.recover(LocalI­ndexShardGateway.java:

15 at
org.elasticsearch.index.gateway.IndexShardGatewayService
$1.run(IndexShardGatewayService.java:179)
16 at

java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:

17 at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:603)
18 at java.lang.Thread.run(Thread.java:636)

[WARN ][cluster.action.shard ] [Sunturion] sending failed shard
for [myindex][0], node[UAmkAQe_TUusC_OvLEtoxg], [P], s[INITIALIZING],
reason [Failed to start shard , message
[IndexShardGatewayRecoveryException[[logstash-2012.03.11][2] shard
allocated for local recovery (post api), should exists, but
doesn't]]]

These errors are repeated for every shards of the index.

Is there an error in my way of using the gateway ?
If not, how can i tell the gateway not to recover the deleted index ?

Thank you.

Johny

On 15 mar, 09:51, Radu Gheorghe radu0gheor...@gmail.com wrote:

Hi,

It's unclear for me: is it possible to do a curl -XDELETE now? If
so,

that should solve it.

If not, what directory did you remove? And what version of ES are
you

using? And how many nodes are in your cluster, with how many shards
and replicas?

I've just deleted
/var/lib/elasticsearch/elasticsearch/nodes/0/indices/

twitter/ on a single-node, and it restarted with no error (and no
index, of course).

Best regards,
Radu

On Mar 14, 8:10 pm, Johny John johnyjo...@gmail.com wrote:

Hello,

I manually deleted an index directory instead of playing a curl -
XDELETE localhost:9200/indexName and i'm know stuck with a
gateway

recovery exception when i whant to restart the server :

failed to start shard :

org.elasticsearch.index.gateway.IndexShardGatewayRecoveryException:

[indexname][2] shard allocated for local recovery (post api),
should

exists, but doesn't

The recevory precedure is looking for a shard that belongs to the
deleted index.

When i restart the server it recreates the index directory and
the

sub

translog dir but everything is empty.

What is the translog file to clean to prevent the recovery ?
Is there any other solution ?

Thank you for.

Johny


(system) #9