Exception when re-balancing shadow replica index shards - unable to determine if root directory exists

I am experimenting shadow replica on my ES 1.7.5 cluster with 3 data nodes.

Following is what I did:

  1. Add following settings to elasticsearch.yml
    node.enable_custom_paths: true
    node.add_id_to_custom_path: false
    path.shared_data: Z:\esindex

Node: Z is mounted drive of the shared files system

  1. Create an index with shadow replica with following setting:
    "index": {
    "data_path": "Z:\esindex",
    "shadow_replicas": true,
    "number_of_shards": 3,
    "number_of_replicas": 2

  2. Insert docs to index

The shadow replica index worked fine for both index and search, until I added a new data node into cluster which triggered shards re-balancing.

I got following exception:

[2016-04-21 01:59:56,555][WARN ][indices.cluster ] [data-vm-3] [[testindex_shadowreplica][0]] marking and sending shard failed due to [failed to create shard]
org.elasticsearch.index.shard.IndexShardCreationException: [testindex_shadowreplica][0] failed to create shard
Caused by: java.nio.file.FileSystemException: Z:\esindex\testindex_shadowreplica\0\index: Unable to determine if root directory exists

What is wrong here?

Hi Youxu,

In the past, when we have seen this, it's an indication that the account that Elasticsearch is running as does not have permission/access to the share that is mounted. Can you verifiy that the right account has ownership and can access/read/write/etc on the share?

Thanks the hint. It sounds reasonable. Will have a look and get back to you.