Exception when re-balancing shadow replica index shards - unable to determine if root directory exists

Elastic Stack Elasticsearch
1

I am experimenting shadow replica on my ES 1.7.5 cluster with 3 data nodes.

Following is what I did:

  1. Add following settings to elasticsearch.yml
    node.enable_custom_paths: true
    node.add_id_to_custom_path: false
    path.shared_data: Z:\esindex

Node: Z is mounted drive of the shared files system

  1. Create an index with shadow replica with following setting:
    "index": {
    "data_path": "Z:\esindex",
    "shadow_replicas": true,
    "number_of_shards": 3,
    "number_of_replicas": 2
    }

  2. Insert docs to index

The shadow replica index worked fine for both index and search, until I added a new data node into cluster which triggered shards re-balancing.

I got following exception:

[2016-04-21 01:59:56,555][WARN ][indices.cluster ] [data-vm-3] [[testindex_shadowreplica][0]] marking and sending shard failed due to [failed to create shard]
org.elasticsearch.index.shard.IndexShardCreationException: [testindex_shadowreplica][0] failed to create shard
Caused by: java.nio.file.FileSystemException: Z:\esindex\testindex_shadowreplica\0\index: Unable to determine if root directory exists

What is wrong here?

2

Hi Youxu,

In the past, when we have seen this, it's an indication that the account that Elasticsearch is running as does not have permission/access to the share that is mounted. Can you verifiy that the right account has ownership and can access/read/write/etc on the share?

3

Thanks the hint. It sounds reasonable. Will have a look and get back to you.