Scenario:
1 master node, no data
3 data nodes
About 10 million documents indexed.
What happened:
It was indexing when master cluster disk became full. So I deleted some files and restarted elasticsearch. Then documents disapeared and didnt show any active shards
Elasticsearch cluster state : Red
Versions:
Elasticsearch : 2.3.0
Kibana : 4.5.1
Cluster health Log:
#curl -s 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "index_app",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 2,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 5,
"unassigned_shards" : 2,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 0.0
}
#curl -s 'http://localhost:9200/_cat/shards'
.kibana 0 p UNASSIGNED
.kibana 0 r UNASSIGNED
vw-docs-prod 1 p INITIALIZING 10.XXX.X.XX3 node-4
vw-docs-prod 3 p INITIALIZING 10.XXX.X.XX3 node-4
vw-docs-prod 2 p INITIALIZING 10.XXX.X.XX2 node-3
vw-docs-prod 4 p INITIALIZING 10.XXX.X.XX3 node-4
vw-docs-prod 0 p INITIALIZING 10.XXX.X.XX2 node-3
#curl -s 'http://localhost:9200/_cat/indices?v'
health status index pri rep docs.count docs.deleted store.size pri.store.size
red open .kibana 1 1
red open vw-docs-prod 5 0
#################################################################
##elasticsearch.yml (data nodes)
cluster.name: cluster_prod
node.name: node-2
node.master: false
node.data: true
network.host: 10.XXX.XX.X11
network.bind_host: 10.XXX.XX.X11
network.publish_to: 10.XXX.XX.X11
index.number_of_shards: 5
index.number_of_replicas: 1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ['masternode.mydomain']
#################################################################
########elasticsearch.yml (master node)
cluster.name: cluster_prod
node.name: node-1-master
node.master: true
node.data: false
network.host: 10.XXX.XX.7
network.bind_host: 10.XXX.XX.7
network.publish_to: 10.XXX.XX.7
index.number_of_shards: 5
index.number_of_replicas: 1
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ['masternode.mydomain']
#################################################################
Is there any way to reconstruct or recover these indices?