Delete returns IndexMissingException but Index creation hangs

Elastic Stack Elasticsearch
1

Hi,
Lately, I am noticing following use case for few of the indexes in the
cluster -

When I issue curl DELETE command, cluster comes up with the response that
index does not exist.

curl -XDELETE 'http://10.0.1.1:9200/du_shnindex_14141_2013_9/'
{"error":"IndexMissingException[[du_shnindex_14141_2013_9]
missing]","status":404}

But when I issue index creation command, it just hangs.
This command just hangs and would eventually timeout.

curl -XPUT 'http://10.0.1.1:9200/du_shnindex_14141_2013_9/'

Why would this happen? I also looked at index folders to check if there are
any index files but I do not see this index in any node.

Please let me know how can I fix it. I am using 0.19.8 version.

Thanks
Ashish

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

There is something blocking the cluster, you have to locate the node having
a problem, and find out the cause by examining the logs. Easiest way to
cope with this is restarting the locked node. There are also many
monitoring tools you can use to watch for the cluster nodes working ok.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks.
I actually restarted the whole cluster and still I am facing same problem
with that particular index.
Please help me understand what can I look for in the logs to know if it is
blocking the cluster.

Another data point is I have 200+ unassigned shards in the cluster at
present and status is Red.
And I see that master node logs are churning out these lines for all those
shards

][WARN ][cluster.action.shard ] [shn-sjc-prod-v-enod4] received shard
failed for [shnindex_11284_2013_9][2], node[1vdpicj6TOubMHOgHjy4PQ], [P],
s[INITIALIZING], reas
on [Failed to start shard, message
[IndexShardGatewayRecoveryException[[shnindex_11284_2013_9][2] shard
allocated for local recovery (post api), should exists, but doesn't]]]

This has been happening since last two weeks now. This is yet another
problem I am facing with the cluster. But the fact that I am not able to
re-create some of those indexes, is blocking me to reindex data from DB.

On Wed, Oct 23, 2013 at 8:27 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

There is something blocking the cluster, you have to locate the node
having a problem, and find out the cause by examining the logs. Easiest way
to cope with this is restarting the locked node. There are also many
monitoring tools you can use to watch for the cluster nodes working ok.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

I thought so the cluster is red.

You should dig in the logs to make sure what happened. Maybe OOM, maybe no
file descriptors, maybe out of disk space, maybe external issues of the OS.
Maybe you find no message at all.

From the recovery message, ES can't find a shard at it's supposed location,
which is fatal, and bails out. It is shnindex_11284_2013_9 shard Id 2 on
node 1vdpicj6TOubMHOgHjy4PQ. If you have replica for this shard, you can
try to remove the broken shard or broken index and hope ES can pick up the
replicated index.

A red cluster can not process index deletions of course.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

If I can't delete an index, then is there any other way to fix the index?
I need to re-populate the data from DB.

On Wed, Oct 23, 2013 at 9:21 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

I thought so the cluster is red.

You should dig in the logs to make sure what happened. Maybe OOM, maybe no
file descriptors, maybe out of disk space, maybe external issues of the OS.
Maybe you find no message at all.

From the recovery message, ES can't find a shard at it's supposed
location, which is fatal, and bails out. It is shnindex_11284_2013_9
shard Id 2 on node 1vdpicj6TOubMHOgHjy4PQ. If you have replica for this
shard, you can try to remove the broken shard or broken index and hope ES
can pick up the replicated index.

A red cluster can not process index deletions of course.

Jörg

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.