# Hundreds of indices missing
**URL:** https://discuss.elastic.co/t/hundreds-of-indices-missing/14531
**Category:** Elasticsearch
**Created:** [November 22, 2013, 1:09am UTC](https://discuss.elastic.co/t/hundreds-of-indices-missing/14531 "2013-11-22T01:09:52Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@Devon\_Crouse](https://discuss.elastic.co/u/Devon_Crouse)
#### Post date: [November 22, 2013, 1:09am UTC](https://discuss.elastic.co/t/hundreds-of-indices-missing/14531/1 "2013-11-22T01:09:52Z")
After restarting the cluster due to some performance issues, we suddenly
had far fewer indices than before. They aren't closed or
recovering/unallocated, they just don't exist according to the API. The
data files still exist on our data nodes and I believe the data is still
there (we had 2 replicas per shard); it's just as if the cluster state was
corrupted somehow. Is there some way to recover or recreate this
state/metadata for the missing indices? We tried turning on verbose
recovery/gateway logging, but still saw no mention of them.
OS: CentOS 6.3 2.6.32-279.5.2.el6.x86\_64 #1 SMP
ES: 0.90.1
Java: Oracle 1.7.0\_17-b02
Topology:
- 3 dedicated masters w/ Zookeeper discovery
- 2 dedicated clients
- 6 dedicated data nodes, each with 2 ES instances (to utilize available
memory and keep heaps \< 32G)
Thanks.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Devon\_Crouse](https://discuss.elastic.co/u/Devon_Crouse)
#### Post date: [November 22, 2013, 8:53pm UTC](https://discuss.elastic.co/t/hundreds-of-indices-missing/14531/2 "2013-11-22T20:53:50Z")
We're having some success recovering with the following. A few of them have
had unrecoverable shards, but most are fine:
#!/usr/bin/env bash
["$#" -eq 1] || { echo "Index required" \>&2; exit 1; }
export idx=$1
# Rename index directories
find /esn/data -type d -name "${idx}" -exec mv {} {}\_bak ; &\> /dev/null
# Recreate empty index
curl -XPUT "[http://localhost:9200/${idx}](http://localhost:9200/%24%7Bidx%7D)" &\> /dev/null
# Allow shards to initialize
sleep 10
# Close the empty index
curl -XPOST "[http://localhost:9200/${idx}/\_close](http://localhost:9200/%24%7Bidx%7D/_close)" &\> /dev/null
# Remove empty index data files
find /esn/data -type d -name "${idx}" -exec rm -rf {} ; &\> /dev/null
# Restore index data
find /esn/data -type d -name "${idx}\_bak" | awk '{ original=$0;
gsub("\_bak",""); system("mv "original" "$0) }'
# Reopen index
curl -XPOST "[http://localhost:9200/${idx}/\_open](http://localhost:9200/%24%7Bidx%7D/_open)" &\> /dev/null
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@Devon\_Crouse](https://discuss.elastic.co/u/Devon_Crouse)
#### Post date: [November 22, 2013, 8:55pm UTC](https://discuss.elastic.co/t/hundreds-of-indices-missing/14531/3 "2013-11-22T20:55:11Z")
To clarify, the above is run on all data nodes concurrently.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 6, 2017, 2:05am UTC](https://discuss.elastic.co/t/hundreds-of-indices-missing/14531/4 "2017-07-06T02:05:14Z")