# Error when using shadow replicas **URL:** \ **Category:** Elasticsearch\ **Created:** [May 7, 2015, 8:59pm UTC](https://discuss.elastic.co/t/error-when-using-shadow-replicas/359 "2015-05-07T20:59:06Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![pweaver](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pweaver/32/44909_2.png) [@pweaver](https://discuss.elastic.co/u/pweaver)\ **Post date:** [May 7, 2015, 8:59pm UTC](https://discuss.elastic.co/t/error-when-using-shadow-replicas/359/1 "2015-05-07T20:59:06Z")
Continuing the discussion from [Avoiding duplicate data and work when using a shared filesystem](https://discuss.elastic.co/t/avoiding-duplicate-data-and-work-when-using-a-shared-filesystem/325/4): I am using elasticsearch-1.5.0. I am following the article at [http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-shadow-replicas.html](http://www.elastic.co/guide/en/elasticsearch/reference/current/indices-shadow-replicas.html). I am able to create an index with shadow\_replica true and number\_of\_replicas 0. However, as soon as I increase number\_of\_replicas to any non-zero number, then I get a flood of this in the logs: ``` [2015-05-07 13:53:29,358][WARN][cluster.action.shard] [lindevelastic1] [test-2015.04.27][0] sending failed shard for [test-2015.04.27][0], node[At9q_V1rQVqXxT8m4EwVPg], [R], s[INITIALIZING], indexUUID [fXq4xQuYQkCEcKyGTJ3ZfA], reason [Failed to start shard, message [RecoveryFailedException[[test-2015.04.27][0]: Recovery failed from [lindevelastic2][LlGB__lyRCOL7cfWIZCE5g][lindevelastic2.vw.rentrak.com][inet[lindevelastic2.vw.rentrak.com/172.26.21.63:9300]]{enable_custom_paths=true, master=false} into [lindevelastic1][At9q_V1rQVqXxT8m4EwVPg][lindevelastic1.vw.rentrak.com][inet[lindevelastic1.vw.rentrak.com/172.26.21.62:9300]]{enable_custom_paths=true, master=true}]; nested: RemoteTransportException[[lindevelastic2][inet[/172.26.21.63:9300]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test-2015.04.27][0] Phase[2] Execution failed]; nested: RemoteTransportException[[lindevelastic1][inet[/172.26.21.62:9300]][internal:index/shard/recovery/prepare_translog]]; nested: EngineCreationFailureException[[test-2015.04.27][0] failed to open index reader]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(default(mmapfs(/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/0/test-2015.04.27/0/index),niofs(/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/0/test-2015.04.27/0/index)), type=MERGE, rate=20.0)]): files: []]; ]] [2015-05-07 13:53:29,358][WARN][cluster.action.shard] [lindevelastic1] [test-2015.04.27][0] received shard failed for [test-2015.04.27][0], node[At9q_V1rQVqXxT8m4EwVPg], [R], s[INITIALIZING], indexUUID [fXq4xQuYQkCEcKyGTJ3ZfA], reason [Failed to start shard, message [RecoveryFailedException[[test-2015.04.27][0]: Recovery failed from [lindevelastic2][LlGB__lyRCOL7cfWIZCE5g][lindevelastic2.vw.rentrak.com][inet[lindevelastic2.vw.rentrak.com/172.26.21.63:9300]]{enable_custom_paths=true, master=false} into [lindevelastic1][At9q_V1rQVqXxT8m4EwVPg][lindevelastic1.vw.rentrak.com][inet[lindevelastic1.vw.rentrak.com/172.26.21.62:9300]]{enable_custom_paths=true, master=true}]; nested: RemoteTransportException[[lindevelastic2][inet[/172.26.21.63:9300]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test-2015.04.27][0] Phase[2] Execution failed]; nested: RemoteTransportException[[lindevelastic1][inet[/172.26.21.62:9300]][internal:index/shard/recovery/prepare_translog]]; nested: EngineCreationFailureException[[test-2015.04.27][0] failed to open index reader]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(default(mmapfs(/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/0/test-2015.04.27/0/index),niofs(/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/0/test-2015.04.27/0/index)), type=MERGE, rate=20.0)]): files: []]; ]] [2015-05-07 13:53:29,396][WARN][cluster.action.shard] [lindevelastic1] [test-2015.04.27][0] received shard failed for [test-2015.04.27][0], node[akGDHoQnQAqBWS3SbOhMoQ], [R], s[INITIALIZING], indexUUID [fXq4xQuYQkCEcKyGTJ3ZfA], reason [Failed to start shard, message [RecoveryFailedException[[test-2015.04.27][0]: Recovery failed from [lindevelastic2][LlGB__lyRCOL7cfWIZCE5g][lindevelastic2.vw.rentrak.com][inet[/172.26.21.63:9300]]{enable_custom_paths=true, master=false} into [lindevelastic4][akGDHoQnQAqBWS3SbOhMoQ][lindevelastic4.vw.rentrak.com][inet[lindevelastic4.vw.rentrak.com/172.26.21.65:9300]]{enable_custom_paths=true, master=false}]; nested: RemoteTransportException[[lindevelastic2][inet[/172.26.21.63:9300]][internal:index/shard/recovery/start_recovery]]; nested: RecoveryEngineException[[test-2015.04.27][0] Phase[2] Execution failed]; nested: RemoteTransportException[[lindevelastic4][inet[/172.26.21.65:9300]][internal:index/shard/recovery/prepare_translog]]; nested: EngineCreationFailureException[[test-2015.04.27][0] failed to open index reader]; nested: IndexNotFoundException[no segments* file found in store(least_used[rate_limited(default(mmapfs(/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/3/test-2015.04.27/0/index),niofs(/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/3/test-2015.04.27/0/index)), type=MERGE, rate=20.0)]): files: []]; ]] [2015-05-07 13:53:29,472][WARN][index.engine] [lindevelastic1] [test-2015.04.27][0] failed to create new reader org.apache.lucene.index.IndexNotFoundException: no segments* file found in store(least_used[rate_limited(default(mmapfs(/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/0/test-2015.04.27/0/index),niofs(/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/0/test-2015.04.27/0/index)), type=MERGE, rate=20.0)]): files: [] ``` And it keeps repeating those errors until I delete the index. ---
**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\ **Post date:** [May 8, 2015, 5:33pm UTC](https://discuss.elastic.co/t/error-when-using-shadow-replicas/359/2 "2015-05-08T17:33:56Z")
Hi pweaver, A couple of questions: - Is `/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/` available on every node in the cluster? If you manually go to the `/mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/0/test-2015.04.27/0/index` directory, can you see a `segments_*` file there? What is the listing of this directory on the second node (the one with the replica)? It sounds like the NFS replication may not be replicating the files that are created on the primary version of the shard? - It looks like you are using NFS, what version of NFS are you using? Currently, Lucene still does not work well with NFS in general due to some file behavior implementation trade-offs that NFS makes. I believe it is better with NFSv4 but still not entirely fixed. ---
**Author:** ![pweaver](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pweaver/32/44909_2.png) [@pweaver](https://discuss.elastic.co/u/pweaver)\ **Post date:** [May 8, 2015, 9:59pm UTC](https://discuss.elastic.co/t/error-when-using-shadow-replicas/359/3 "2015-05-08T21:59:28Z")
Yes, the base directory (/mnt/nfs/vol\_na3\_lindevelastic\_nfs/shared/elasticsearch/shadow-indices/) is available on each node. When I first create the index (with no replicas), then a directory gets created with the segments in it. For example, if node 4 gets the primary shard, then I see this: ``` ls /mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/4/test/0/index/ segments_1 segments.gen write.lock ``` If I then increase number\_of\_replicas to 1, the errors start showing up in the logs, and this is what I see in the directory of another node, which was supposed to get the replica: ``` ls /mnt/nfs/vol_na3_lindevelastic_nfs/shared/elasticsearch/shadow-indices/2/test/0/index/ ``` The directory is empty. Of course, the point of shadow replicas is that the data shouldn't be duplicated, right? Perhaps I should set node.add\_id\_to\_custom\_path to false so that they use the exact same directory? ---
**Author:** ![dakrone](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dakrone/32/23351_2.png) [@dakrone](https://discuss.elastic.co/u/dakrone)\ **Post date:** [May 8, 2015, 10:29pm UTC](https://discuss.elastic.co/t/error-when-using-shadow-replicas/359/4 "2015-05-08T22:29:41Z")
> [@pweaver](#): > > Perhaps I should set node.add\_id\_to\_custom\_path to false so that they use the exact same directory? Yes, you should do this if you are running nodes on the same machine. ---
**Author:** ![pweaver](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pweaver/32/44909_2.png) [@pweaver](https://discuss.elastic.co/u/pweaver)\ **Post date:** [May 8, 2015, 10:42pm UTC](https://discuss.elastic.co/t/error-when-using-shadow-replicas/359/5 "2015-05-08T22:42:56Z")
> [@dakrone](#): > > Yes, you should do this if you are running nodes on the same machine. Or if I'm sharing an NFS mount across multiple machines, right? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:14am UTC](https://discuss.elastic.co/t/error-when-using-shadow-replicas/359/6 "2017-07-06T00:14:56Z")