Shadow replica's issue

Elastic Stack Elasticsearch
1

Hello,

I'm confused on how to use shadow replica's.

My setup is as follows. 3 node cluster. ES 2.1.0

cluster.name: elksetuppoc
node.name: elastic6
index.number_of_shards: 5
index.number_of_replicas: 1
gateway.recover_after_nodes: 2
gateway.recover_after_time: 5m
gateway.expected_nodes: 3
bootstrap.mlockall: true
discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["node1", "node2","node3"]
network.host: 0.0.0.0
node.add_id_to_custom_path: false
path.shared_data: /srv/elasticsearch

/srv/elasticsearch is an NFS share that is ** r/w on all nodes**

UPDATE: 30/11 I tested without NFS and no shared storage and I have the same issue.

My goal is to use shadow replicas so I can move older data to this NFS share. Either I'm doing something wrong. Or I have misunderstood something. Or I have git a bug. :wink:

I change the number of replicas to 0 because data will be on shared file system

curl -XPUT http://node1:9200/logstash-2015.11.07/_settings -d '
{
  "index": {
  "number_of_replicas": 0
  }
}
'

Up and till here I can see on the backend that the number of replicas is reduced.

I close the index because I can't change data_path when it is open

curl -XPOST http://node1:9200/logstash-2015.11.07/_close

that works

I change datapath, and enable shadow_replicas

curl -XPUT http://node1:9200/logstash-2015.11.07/_settings -d '
{
  "index": {
  "data_path": "/srv/elasticsearch",
  "shadow_replicas": true
  }
}
'
curl -XPOST http://node1:9200/logstash-2015.11.07/_close

That also succeeds, however on disk I see

root@node1:~# find /srv/elasticsearch/logstash-2015.11.07/
/srv/elasticsearch/logstash-2015.11.07/
/srv/elasticsearch/logstash-2015.11.07/2
/srv/elasticsearch/logstash-2015.11.07/2/index
/srv/elasticsearch/logstash-2015.11.07/0
/srv/elasticsearch/logstash-2015.11.07/0/index
/srv/elasticsearch/logstash-2015.11.07/4
/srv/elasticsearch/logstash-2015.11.07/4/index
/srv/elasticsearch/logstash-2015.11.07/1
/srv/elasticsearch/logstash-2015.11.07/1/index
/srv/elasticsearch/logstash-2015.11.07/3
/srv/elasticsearch/logstash-2015.11.07/3/index

And those dirs are emtpy

The data is still under /var/lib/elasticsearch/

root@node1:~# du -hs /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/*
112K    /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/3
272K    /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/4
8.0K    /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/_state

root@node2:~# du -hs /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/*
88K     /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/0
88K     /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/1
72K     /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/2
8.0K    /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/_state

root@node3:~# du -hs /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/*
72K     /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/0
8.0K    /var/lib/elasticsearch/elksetuppoc/nodes/0/indices/logstash-2015.11.07/_state

When I use kibana to search for the data it doesn't show any entries for those indexes. The others that weren't moved are displayed correctly.

Am I missing something in my setup or did I misunderstand something?

Thanks in advance,