# Store location is not shared between node **URL:** \ **Category:** Elasticsearch\ **Created:** [February 4, 2015, 12:18am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967 "2015-02-04T00:18:39Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Andre\_Hansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_hansel/32/934_2.png) [@Andre\_Hansel](https://discuss.elastic.co/u/Andre_Hansel)\ **Post date:** [February 4, 2015, 12:18am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/1 "2015-02-04T00:18:39Z")
I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store). When I try to create the repository, I get an error 500: error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [\_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][\_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; " I can see with inotify that it creates two files like tests-yOvEwIjQQP6rs\_xqf-iFAQ-master tests-yOvEwIjQQP6rs\_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q But those two files are removed immediately after, before they even arrive on the other node. How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"? In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 4, 2015, 5:44am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/2 "2015-02-04T05:44:53Z")
Elasticsearch does not really wait. I mean that once a file is created on a shared FS it should be visible immediately on other nodes that share the same FS. Is your FS really shared on both nodes using the same mount point? David > Le 4 févr. 2015 à 01:18, André Hänsel [andre@webkr.de](mailto:andre@webkr.de) a écrit : > > I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store). > > When I try to create the repository, I get an error 500: > error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [\_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][\_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; " > > I can see with inotify that it creates two files like > tests-yOvEwIjQQP6rs\_xqf-iFAQ-master > tests-yOvEwIjQQP6rs\_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q > > But those two files are removed immediately after, before they even arrive on the other node. > > How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"? > > ## In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course. > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/FDED528F-5448-4922-AF22-05D7C7D9B490%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/FDED528F-5448-4922-AF22-05D7C7D9B490%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 4, 2015, 5:59am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/3 "2015-02-04T05:59:24Z")
> In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course. Replication is also about creating a copy of the data. So if a node dies or a local disk crashes you will have a copy somewhere. Also, shared FS means to me Network latency. I believe you won't have the same performance as you can have with local drives. David > Le 4 févr. 2015 à 01:18, André Hänsel [andre@webkr.de](mailto:andre@webkr.de) a écrit : > > I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store). > > When I try to create the repository, I get an error 500: > error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [\_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][\_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; " > > I can see with inotify that it creates two files like > tests-yOvEwIjQQP6rs\_xqf-iFAQ-master > tests-yOvEwIjQQP6rs\_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q > > But those two files are removed immediately after, before they even arrive on the other node. > > How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"? > > ## In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course. > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/F392EFFA-6649-47B3-9826-5B9C2423CDB8%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/F392EFFA-6649-47B3-9826-5B9C2423CDB8%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andre\_Hansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_hansel/32/934_2.png) [@Andre\_Hansel](https://discuss.elastic.co/u/Andre_Hansel)\ **Post date:** [February 4, 2015, 6:04am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/4 "2015-02-04T06:04:35Z")
Is this snapshot type just for nodes that run on the same machine? The Elasticsearch guide didn't really say anything about requirements to that filesystem in terms of atomicity, delay, locking, etc. If Elasticsearch doesn't wait, you would need a shared filesystem where the fopen call blocks until it is replicated on all nodes. That would mean DRBD in Protocol A ([LINBIT Software Download Page For LINSTOR And DRBD Linux Driver](https://drbd.linbit.com/users-guide/s-replication-protocols.html)) wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem based on S3, FTP, SSH, etc. On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote: > Elasticsearch does not really wait. I mean that once a file is created on > a shared FS it should be visible immediately on other nodes that share the > same FS. > > Is your FS really shared on both nodes using the same mount point? > > David > > Le 4 févr. 2015 à 01:18, André Hänsel \<[an...@webkr.de](mailto:an...@webkr.de) \\> a > écrit : > > I'm trying to set up a snapshot repository to get a consistent backup (in > order to use elasticsearch as primary data store). > > When I try to create the repository, I get an error 500: > error: "RemoteTransportException[[Jack > Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: > RepositoryVerificationException[[hurz] [\_kKn5dODRzmPOhi05U59yg, > 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; > nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is > not shared between node > [[Box][\_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and > the master node]; ']]]; " > > I can see with inotify that it creates two files like > tests-yOvEwIjQQP6rs\_xqf-iFAQ-master > tests-yOvEwIjQQP6rs\_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q > > But those two files are removed immediately after, before they even arrive > on the other node. > > How long does elasticsearch wait for the files to appear on the other node > before it decides that the store location "is not shared"? > > In general I think having to have a shared filesystem for backup takes > away some of the advantage of elasticsearch replicating itself. If I have a > shared filesystem anyway, I could just put the index there. For indexes > that are not continuously updated of course. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 4, 2015, 6:11am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/5 "2015-02-04T06:11:51Z")
Shared FS needs to support Atomic moves. This is something checked when you create the repo and I think it could give you the message you have seen. [https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108](https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108) David > Le 4 févr. 2015 à 07:04, André Hänsel [andre@webkr.de](mailto:andre@webkr.de) a écrit : > > Is this snapshot type just for nodes that run on the same machine? The Elasticsearch guide didn't really say anything about requirements to that filesystem in terms of atomicity, delay, locking, etc. > > If Elasticsearch doesn't wait, you would need a shared filesystem where the fopen call blocks until it is replicated on all nodes. That would mean DRBD in Protocol A ([LINBIT Software Download Page For LINSTOR And DRBD Linux Driver](https://drbd.linbit.com/users-guide/s-replication-protocols.html)) wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem based on S3, FTP, SSH, etc. > > > On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote: > > Elasticsearch does not really wait. I mean that once a file is created on a shared FS it should be visible immediately on other nodes that share the same FS. > > > > Is your FS really shared on both nodes using the same mount point? > > > > David > > > > > Le 4 févr. 2015 à 01:18, André Hänsel [an...@webkr.de](mailto:an...@webkr.de) a écrit : > > > > > > I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store). > > > > > > When I try to create the repository, I get an error 500: > > > error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [\_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][\_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; " > > > > > > I can see with inotify that it creates two files like > > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-master > > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q > > > > > > But those two files are removed immediately after, before they even arrive on the other node. > > > > > > How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"? > > > > > > ## In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course. > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5FA24D60-15E9-45E5-AB68-3E03DA57A4D0%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/5FA24D60-15E9-45E5-AB68-3E03DA57A4D0%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andre\_Hansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_hansel/32/934_2.png) [@Andre\_Hansel](https://discuss.elastic.co/u/Andre_Hansel)\ **Post date:** [February 4, 2015, 6:52am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/6 "2015-02-04T06:52:24Z")
Ok, but that has nothing to do with the write being synchronous (blocking) or asynchronous, does it? On Wednesday, February 4, 2015 at 7:12:17 AM UTC+1, David Pilato wrote: > Shared FS needs to support Atomic moves. This is something checked when > you create the repo and I think it could give you the message you have seen. > > [https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108](https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108) > > David > > Le 4 févr. 2015 à 07:04, André Hänsel \<[an...@webkr.de](mailto:an...@webkr.de) \\> a > écrit : > > Is this snapshot type just for nodes that run on the same machine? The > Elasticsearch guide didn't really say anything about requirements to that > filesystem in terms of atomicity, delay, locking, etc. > > If Elasticsearch doesn't wait, you would need a shared filesystem where > the fopen call blocks until it is replicated on all nodes. That would mean > DRBD in Protocol A ( > [LINBIT Software Download Page For LINSTOR And DRBD Linux Driver](https://drbd.linbit.com/users-guide/s-replication-protocols.html)) > wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the > point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem > based on S3, FTP, SSH, etc. > > On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote: > > > Elasticsearch does not really wait. I mean that once a file is created on > > a shared FS it should be visible immediately on other nodes that share the > > same FS. > > > > Is your FS really shared on both nodes using the same mount point? > > > > David > > > > Le 4 févr. 2015 à 01:18, André Hänsel [an...@webkr.de](mailto:an...@webkr.de) a écrit : > > > > I'm trying to set up a snapshot repository to get a consistent backup (in > > order to use elasticsearch as primary data store). > > > > When I try to create the repository, I get an error 500: > > error: "RemoteTransportException[[Jack > > Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: > > RepositoryVerificationException[[hurz] [\_kKn5dODRzmPOhi05U59yg, > > 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; > > nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is > > not shared between node > > [[Box][\_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and > > the master node]; ']]]; " > > > > I can see with inotify that it creates two files like > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-master > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q > > > > But those two files are removed immediately after, before they even > > arrive on the other node. > > > > How long does elasticsearch wait for the files to appear on the other > > node before it decides that the store location "is not shared"? > > > > In general I think having to have a shared filesystem for backup takes > > away some of the advantage of elasticsearch replicating itself. If I have a > > shared filesystem anyway, I could just put the index there. For indexes > > that are not continuously updated of course. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 4, 2015, 7:10am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/7 "2015-02-04T07:10:47Z")
When we check the repository, we wait for the move to be done before trying to read the file from another node. BTW you can pass ?verify=false when you create the repo so it won't be checked. David > Le 4 févr. 2015 à 07:52, André Hänsel [andre@webkr.de](mailto:andre@webkr.de) a écrit : > > Ok, but that has nothing to do with the write being synchronous (blocking) or asynchronous, does it? > > > On Wednesday, February 4, 2015 at 7:12:17 AM UTC+1, David Pilato wrote: > > Shared FS needs to support Atomic moves. This is something checked when you create the repo and I think it could give you the message you have seen. > > > > [https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108](https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108) > > > > David > > > > > Le 4 févr. 2015 à 07:04, André Hänsel [an...@webkr.de](mailto:an...@webkr.de) a écrit : > > > > > > Is this snapshot type just for nodes that run on the same machine? The Elasticsearch guide didn't really say anything about requirements to that filesystem in terms of atomicity, delay, locking, etc. > > > > > > If Elasticsearch doesn't wait, you would need a shared filesystem where the fopen call blocks until it is replicated on all nodes. That would mean DRBD in Protocol A ([LINBIT Software Download Page For LINSTOR And DRBD Linux Driver](https://drbd.linbit.com/users-guide/s-replication-protocols.html)) wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem based on S3, FTP, SSH, etc. > > > > > > > On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote: > > > > Elasticsearch does not really wait. I mean that once a file is created on a shared FS it should be visible immediately on other nodes that share the same FS. > > > > > > > > Is your FS really shared on both nodes using the same mount point? > > > > > > > > David > > > > > > > > > Le 4 févr. 2015 à 01:18, André Hänsel [an...@webkr.de](mailto:an...@webkr.de) a écrit : > > > > > > > > > > I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store). > > > > > > > > > > When I try to create the repository, I get an error 500: > > > > > error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [\_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][\_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; " > > > > > > > > > > I can see with inotify that it creates two files like > > > > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-master > > > > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q > > > > > > > > > > But those two files are removed immediately after, before they even arrive on the other node. > > > > > > > > > > How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"? > > > > > > > > > > ## In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course. > > > > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com). > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2C60DC6C-C6C4-4126-812C-E5345A00A79C%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/2C60DC6C-C6C4-4126-812C-E5345A00A79C%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Andre\_Hansel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andre_hansel/32/934_2.png) [@Andre\_Hansel](https://discuss.elastic.co/u/Andre_Hansel)\ **Post date:** [February 4, 2015, 10:06pm UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/8 "2015-02-04T22:06:35Z")
That is indeed helpful. 🙂 However, I just noticed that it's not even necessary - the repository is still created, despite the error 500 returned. On Wednesday, February 4, 2015 at 8:11:12 AM UTC+1, David Pilato wrote: > When we check the repository, we wait for the move to be done before > trying to read the file from another node. > > BTW you can pass ?verify=false when you create the repo so it won't be > checked. > > David > > Le 4 févr. 2015 à 07:52, André Hänsel \<[an...@webkr.de](mailto:an...@webkr.de) \\> a > écrit : > > Ok, but that has nothing to do with the write being synchronous (blocking) > or asynchronous, does it? > > On Wednesday, February 4, 2015 at 7:12:17 AM UTC+1, David Pilato wrote: > > > Shared FS needs to support Atomic moves. This is something checked when > > you create the repo and I think it could give you the message you have seen. > > > > [https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108](https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108) > > > > David > > > > Le 4 févr. 2015 à 07:04, André Hänsel [an...@webkr.de](mailto:an...@webkr.de) a écrit : > > > > Is this snapshot type just for nodes that run on the same machine? The > > Elasticsearch guide didn't really say anything about requirements to that > > filesystem in terms of atomicity, delay, locking, etc. > > > > If Elasticsearch doesn't wait, you would need a shared filesystem where > > the fopen call blocks until it is replicated on all nodes. That would mean > > DRBD in Protocol A ( > > [LINBIT Software Download Page For LINSTOR And DRBD Linux Driver](https://drbd.linbit.com/users-guide/s-replication-protocols.html)) > > wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the > > point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem > > based on S3, FTP, SSH, etc. > > > > On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote: > > > > > Elasticsearch does not really wait. I mean that once a file is created > > > on a shared FS it should be visible immediately on other nodes that share > > > the same FS. > > > > > > Is your FS really shared on both nodes using the same mount point? > > > > > > David > > > > > > Le 4 févr. 2015 à 01:18, André Hänsel [an...@webkr.de](mailto:an...@webkr.de) a écrit : > > > > > > I'm trying to set up a snapshot repository to get a consistent backup > > > (in order to use elasticsearch as primary data store). > > > > > > When I try to create the repository, I get an error 500: > > > error: "RemoteTransportException[[Jack > > > Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: > > > RepositoryVerificationException[[hurz] [\_kKn5dODRzmPOhi05U59yg, > > > 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; > > > nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is > > > not shared between node > > > [[Box][\_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and > > > the master node]; ']]]; " > > > > > > I can see with inotify that it creates two files like > > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-master > > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q > > > > > > But those two files are removed immediately after, before they even > > > arrive on the other node. > > > > > > How long does elasticsearch wait for the files to appear on the other > > > node before it decides that the store location "is not shared"? > > > > > > In general I think having to have a shared filesystem for backup takes > > > away some of the advantage of elasticsearch replicating itself. If I have a > > > shared filesystem anyway, I could just put the index there. For indexes > > > that are not continuously updated of course. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups > > > "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an > > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f6a9f812-6656-46e0-ab5f-f212671b29d1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f6a9f812-6656-46e0-ab5f-f212671b29d1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [February 5, 2015, 6:06am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/9 "2015-02-05T06:06:59Z")
Really? This should not happen I think. Could you reproduce it and open an issue with a scenario we can use to reproduce this? David > Le 4 févr. 2015 à 23:06, André Hänsel [andre@webkr.de](mailto:andre@webkr.de) a écrit : > > That is indeed helpful. 🙂 However, I just noticed that it's not even necessary - the repository is still created, despite the error 500 returned. > > > On Wednesday, February 4, 2015 at 8:11:12 AM UTC+1, David Pilato wrote: > > When we check the repository, we wait for the move to be done before trying to read the file from another node. > > > > BTW you can pass ?verify=false when you create the repo so it won't be checked. > > > > David > > > > > Le 4 févr. 2015 à 07:52, André Hänsel [an...@webkr.de](mailto:an...@webkr.de) a écrit : > > > > > > Ok, but that has nothing to do with the write being synchronous (blocking) or asynchronous, does it? > > > > > > > On Wednesday, February 4, 2015 at 7:12:17 AM UTC+1, David Pilato wrote: > > > > Shared FS needs to support Atomic moves. This is something checked when you create the repo and I think it could give you the message you have seen. > > > > > > > > [https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108](https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108) > > > > > > > > David > > > > > > > > > Le 4 févr. 2015 à 07:04, André Hänsel [an...@webkr.de](mailto:an...@webkr.de) a écrit : > > > > > > > > > > Is this snapshot type just for nodes that run on the same machine? The Elasticsearch guide didn't really say anything about requirements to that filesystem in terms of atomicity, delay, locking, etc. > > > > > > > > > > If Elasticsearch doesn't wait, you would need a shared filesystem where the fopen call blocks until it is replicated on all nodes. That would mean DRBD in Protocol A ([LINBIT Software Download Page For LINSTOR And DRBD Linux Driver](https://drbd.linbit.com/users-guide/s-replication-protocols.html)) wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem based on S3, FTP, SSH, etc. > > > > > > > > > > > On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote: > > > > > > Elasticsearch does not really wait. I mean that once a file is created on a shared FS it should be visible immediately on other nodes that share the same FS. > > > > > > > > > > > > Is your FS really shared on both nodes using the same mount point? > > > > > > > > > > > > David > > > > > > > > > > > > > Le 4 févr. 2015 à 01:18, André Hänsel [an...@webkr.de](mailto:an...@webkr.de) a écrit : > > > > > > > > > > > > > > I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store). > > > > > > > > > > > > > > When I try to create the repository, I get an error 500: > > > > > > > error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [\_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][\_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; " > > > > > > > > > > > > > > I can see with inotify that it creates two files like > > > > > > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-master > > > > > > > tests-yOvEwIjQQP6rs\_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q > > > > > > > > > > > > > > But those two files are removed immediately after, before they even arrive on the other node. > > > > > > > > > > > > > > How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"? > > > > > > > > > > > > > > ## In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course. > > > > > > > > > > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com). > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com). > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com). > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f6a9f812-6656-46e0-ab5f-f212671b29d1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f6a9f812-6656-46e0-ab5f-f212671b29d1%40googlegroups.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/36BBB027-811E-44BD-A9C5-36F9461F468B%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/36BBB027-811E-44BD-A9C5-36F9461F468B%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:34am UTC](https://discuss.elastic.co/t/store-location-is-not-shared-between-node/21967/10 "2017-07-06T00:34:53Z")