Store location is not shared between node

Andre_Hansel · February 4, 2015, 12:18am

I'm trying to set up a snapshot repository to get a consistent backup (in
order to use elasticsearch as primary data store).

When I try to create the repository, I get an error 500:
error: "RemoteTransportException[[Jack
Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested:
RepositoryVerificationException[[hurz] [_kKn5dODRzmPOhi05U59yg,
'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is
not shared between node
[[Box][_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and
the master node]; ']]]; "

I can see with inotify that it creates two files like
tests-yOvEwIjQQP6rs_xqf-iFAQ-master
tests-yOvEwIjQQP6rs_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q

But those two files are removed immediately after, before they even arrive
on the other node.

How long does elasticsearch wait for the files to appear on the other node
before it decides that the store location "is not shared"?

In general I think having to have a shared filesystem for backup takes away
some of the advantage of elasticsearch replicating itself. If I have a
shared filesystem anyway, I could just put the index there. For indexes
that are not continuously updated of course.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · February 4, 2015, 5:44am

Elasticsearch does not really wait. I mean that once a file is created on a shared FS it should be visible immediately on other nodes that share the same FS.

Is your FS really shared on both nodes using the same mount point?

David

Le 4 févr. 2015 à 01:18, André Hänsel andre@webkr.de a écrit :

I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store).

When I try to create the repository, I get an error 500:
error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; "

I can see with inotify that it creates two files like
tests-yOvEwIjQQP6rs_xqf-iFAQ-master
tests-yOvEwIjQQP6rs_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q

But those two files are removed immediately after, before they even arrive on the other node.

How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"?

In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/FDED528F-5448-4922-AF22-05D7C7D9B490%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

dadoonet · February 4, 2015, 5:59am

In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course.

Replication is also about creating a copy of the data. So if a node dies or a local disk crashes you will have a copy somewhere.
Also, shared FS means to me Network latency. I believe you won't have the same performance as you can have with local drives.

David

Le 4 févr. 2015 à 01:18, André Hänsel andre@webkr.de a écrit :

I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store).

When I try to create the repository, I get an error 500:
error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; "

I can see with inotify that it creates two files like
tests-yOvEwIjQQP6rs_xqf-iFAQ-master
tests-yOvEwIjQQP6rs_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q

But those two files are removed immediately after, before they even arrive on the other node.

How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"?

In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/F392EFFA-6649-47B3-9826-5B9C2423CDB8%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Andre_Hansel · February 4, 2015, 6:04am

Is this snapshot type just for nodes that run on the same machine? The
Elasticsearch guide didn't really say anything about requirements to that
filesystem in terms of atomicity, delay, locking, etc.

If Elasticsearch doesn't wait, you would need a shared filesystem where the
fopen call blocks until it is replicated on all nodes. That would mean DRBD
in Protocol A
(LINBIT Software Download Page For LINSTOR And DRBD Linux Driver) wouldn't
work. (Doesn't won't for dual-primary anyway, but that's not the point
here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem based
on S3, FTP, SSH, etc.

On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote:

Elasticsearch does not really wait. I mean that once a file is created on
a shared FS it should be visible immediately on other nodes that share the
same FS.

Is your FS really shared on both nodes using the same mount point?

David

Le 4 févr. 2015 à 01:18, André Hänsel <an...@webkr.de <javascript:>> a
écrit :

I'm trying to set up a snapshot repository to get a consistent backup (in
order to use elasticsearch as primary data store).

When I try to create the repository, I get an error 500:
error: "RemoteTransportException[[Jack
Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested:
RepositoryVerificationException[[hurz] [_kKn5dODRzmPOhi05U59yg,
'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is
not shared between node
[[Box][_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and
the master node]; ']]]; "

I can see with inotify that it creates two files like
tests-yOvEwIjQQP6rs_xqf-iFAQ-master
tests-yOvEwIjQQP6rs_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q

But those two files are removed immediately after, before they even arrive
on the other node.

How long does elasticsearch wait for the files to appear on the other node
before it decides that the store location "is not shared"?

In general I think having to have a shared filesystem for backup takes
away some of the advantage of elasticsearch replicating itself. If I have a
shared filesystem anyway, I could just put the index there. For indexes
that are not continuously updated of course.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · February 4, 2015, 6:11am

Shared FS needs to support Atomic moves. This is something checked when you create the repo and I think it could give you the message you have seen.

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108

David

Le 4 févr. 2015 à 07:04, André Hänsel andre@webkr.de a écrit :

Is this snapshot type just for nodes that run on the same machine? The Elasticsearch guide didn't really say anything about requirements to that filesystem in terms of atomicity, delay, locking, etc.

If Elasticsearch doesn't wait, you would need a shared filesystem where the fopen call blocks until it is replicated on all nodes. That would mean DRBD in Protocol A (LINBIT Software Download Page For LINSTOR And DRBD Linux Driver) wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem based on S3, FTP, SSH, etc.

On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote:
Elasticsearch does not really wait. I mean that once a file is created on a shared FS it should be visible immediately on other nodes that share the same FS.

Is your FS really shared on both nodes using the same mount point?

David

Le 4 févr. 2015 à 01:18, André Hänsel an...@webkr.de a écrit :

I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store).

When I try to create the repository, I get an error 500:
error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; "

I can see with inotify that it creates two files like
tests-yOvEwIjQQP6rs_xqf-iFAQ-master
tests-yOvEwIjQQP6rs_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q

But those two files are removed immediately after, before they even arrive on the other node.

How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"?

In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5FA24D60-15E9-45E5-AB68-3E03DA57A4D0%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Andre_Hansel · February 4, 2015, 6:52am

Ok, but that has nothing to do with the write being synchronous (blocking)
or asynchronous, does it?

On Wednesday, February 4, 2015 at 7:12:17 AM UTC+1, David Pilato wrote:

Shared FS needs to support Atomic moves. This is something checked when
you create the repo and I think it could give you the message you have seen.

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108

David

Le 4 févr. 2015 à 07:04, André Hänsel <an...@webkr.de <javascript:>> a
écrit :

Is this snapshot type just for nodes that run on the same machine? The
Elasticsearch guide didn't really say anything about requirements to that
filesystem in terms of atomicity, delay, locking, etc.

If Elasticsearch doesn't wait, you would need a shared filesystem where
the fopen call blocks until it is replicated on all nodes. That would mean
DRBD in Protocol A (
LINBIT Software Download Page For LINSTOR And DRBD Linux Driver)
wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the
point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem
based on S3, FTP, SSH, etc.

On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote:

Elasticsearch does not really wait. I mean that once a file is created on
a shared FS it should be visible immediately on other nodes that share the
same FS.

Is your FS really shared on both nodes using the same mount point?

David

Le 4 févr. 2015 à 01:18, André Hänsel an...@webkr.de a écrit :

I'm trying to set up a snapshot repository to get a consistent backup (in
order to use elasticsearch as primary data store).

When I try to create the repository, I get an error 500:
error: "RemoteTransportException[[Jack
Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested:
RepositoryVerificationException[[hurz] [_kKn5dODRzmPOhi05U59yg,
'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is
not shared between node
[[Box][_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and
the master node]; ']]]; "

I can see with inotify that it creates two files like
tests-yOvEwIjQQP6rs_xqf-iFAQ-master
tests-yOvEwIjQQP6rs_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q

But those two files are removed immediately after, before they even
arrive on the other node.

How long does elasticsearch wait for the files to appear on the other
node before it decides that the store location "is not shared"?

In general I think having to have a shared filesystem for backup takes
away some of the advantage of elasticsearch replicating itself. If I have a
shared filesystem anyway, I could just put the index there. For indexes
that are not continuously updated of course.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · February 4, 2015, 7:10am

When we check the repository, we wait for the move to be done before trying to read the file from another node.

BTW you can pass ?verify=false when you create the repo so it won't be checked.

David

Le 4 févr. 2015 à 07:52, André Hänsel andre@webkr.de a écrit :

Ok, but that has nothing to do with the write being synchronous (blocking) or asynchronous, does it?

On Wednesday, February 4, 2015 at 7:12:17 AM UTC+1, David Pilato wrote:
Shared FS needs to support Atomic moves. This is something checked when you create the repo and I think it could give you the message you have seen.

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108

David

Le 4 févr. 2015 à 07:04, André Hänsel an...@webkr.de a écrit :

Is this snapshot type just for nodes that run on the same machine? The Elasticsearch guide didn't really say anything about requirements to that filesystem in terms of atomicity, delay, locking, etc.

If Elasticsearch doesn't wait, you would need a shared filesystem where the fopen call blocks until it is replicated on all nodes. That would mean DRBD in Protocol A (LINBIT Software Download Page For LINSTOR And DRBD Linux Driver) wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem based on S3, FTP, SSH, etc.

On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote:
Elasticsearch does not really wait. I mean that once a file is created on a shared FS it should be visible immediately on other nodes that share the same FS.

Is your FS really shared on both nodes using the same mount point?

David

Le 4 févr. 2015 à 01:18, André Hänsel an...@webkr.de a écrit :

I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store).

When I try to create the repository, I get an error 500:
error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; "

I can see with inotify that it creates two files like
tests-yOvEwIjQQP6rs_xqf-iFAQ-master
tests-yOvEwIjQQP6rs_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q

But those two files are removed immediately after, before they even arrive on the other node.

How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"?

In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2C60DC6C-C6C4-4126-812C-E5345A00A79C%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.

Andre_Hansel · February 4, 2015, 10:06pm

That is indeed helpful. However, I just noticed that it's not even
necessary - the repository is still created, despite the error 500 returned.

On Wednesday, February 4, 2015 at 8:11:12 AM UTC+1, David Pilato wrote:

When we check the repository, we wait for the move to be done before
trying to read the file from another node.

BTW you can pass ?verify=false when you create the repo so it won't be
checked.

David

Le 4 févr. 2015 à 07:52, André Hänsel <an...@webkr.de <javascript:>> a
écrit :

Ok, but that has nothing to do with the write being synchronous (blocking)
or asynchronous, does it?

On Wednesday, February 4, 2015 at 7:12:17 AM UTC+1, David Pilato wrote:

Shared FS needs to support Atomic moves. This is something checked when
you create the repo and I think it could give you the message you have seen.

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108

David

Le 4 févr. 2015 à 07:04, André Hänsel an...@webkr.de a écrit :

Is this snapshot type just for nodes that run on the same machine? The
Elasticsearch guide didn't really say anything about requirements to that
filesystem in terms of atomicity, delay, locking, etc.

If Elasticsearch doesn't wait, you would need a shared filesystem where
the fopen call blocks until it is replicated on all nodes. That would mean
DRBD in Protocol A (
LINBIT Software Download Page For LINSTOR And DRBD Linux Driver)
wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the
point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem
based on S3, FTP, SSH, etc.

On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote:

Elasticsearch does not really wait. I mean that once a file is created
on a shared FS it should be visible immediately on other nodes that share
the same FS.

Is your FS really shared on both nodes using the same mount point?

David

Le 4 févr. 2015 à 01:18, André Hänsel an...@webkr.de a écrit :

I'm trying to set up a snapshot repository to get a consistent backup
(in order to use elasticsearch as primary data store).

When I try to create the repository, I get an error 500:
error: "RemoteTransportException[[Jack
Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested:
RepositoryVerificationException[[hurz] [_kKn5dODRzmPOhi05U59yg,
'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]];
nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is
not shared between node
[[Box][_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and
the master node]; ']]]; "

I can see with inotify that it creates two files like
tests-yOvEwIjQQP6rs_xqf-iFAQ-master
tests-yOvEwIjQQP6rs_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q

But those two files are removed immediately after, before they even
arrive on the other node.

How long does elasticsearch wait for the files to appear on the other
node before it decides that the store location "is not shared"?

In general I think having to have a shared filesystem for backup takes
away some of the advantage of elasticsearch replicating itself. If I have a
shared filesystem anyway, I could just put the index there. For indexes
that are not continuously updated of course.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f6a9f812-6656-46e0-ab5f-f212671b29d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

dadoonet · February 5, 2015, 6:06am

Really? This should not happen I think. Could you reproduce it and open an issue with a scenario we can use to reproduce this?

David

Le 4 févr. 2015 à 23:06, André Hänsel andre@webkr.de a écrit :

That is indeed helpful. However, I just noticed that it's not even necessary - the repository is still created, despite the error 500 returned.

On Wednesday, February 4, 2015 at 8:11:12 AM UTC+1, David Pilato wrote:
When we check the repository, we wait for the move to be done before trying to read the file from another node.

BTW you can pass ?verify=false when you create the repo so it won't be checked.

David

Le 4 févr. 2015 à 07:52, André Hänsel an...@webkr.de a écrit :

Ok, but that has nothing to do with the write being synchronous (blocking) or asynchronous, does it?

On Wednesday, February 4, 2015 at 7:12:17 AM UTC+1, David Pilato wrote:
Shared FS needs to support Atomic moves. This is something checked when you create the repo and I think it could give you the message you have seen.

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/common/blobstore/fs/FsBlobContainer.java#L108

David

Le 4 févr. 2015 à 07:04, André Hänsel an...@webkr.de a écrit :

Is this snapshot type just for nodes that run on the same machine? The Elasticsearch guide didn't really say anything about requirements to that filesystem in terms of atomicity, delay, locking, etc.

If Elasticsearch doesn't wait, you would need a shared filesystem where the fopen call blocks until it is replicated on all nodes. That would mean DRBD in Protocol A (LINBIT Software Download Page For LINSTOR And DRBD Linux Driver) wouldn't work. (Doesn't won't for dual-primary anyway, but that's not the point here.) Lsyncd or csync2 wouldn't work either. Or any FUSE filesystem based on S3, FTP, SSH, etc.

On Wednesday, February 4, 2015 at 6:45:18 AM UTC+1, David Pilato wrote:
Elasticsearch does not really wait. I mean that once a file is created on a shared FS it should be visible immediately on other nodes that share the same FS.

Is your FS really shared on both nodes using the same mount point?

David

Le 4 févr. 2015 à 01:18, André Hänsel an...@webkr.de a écrit :

I'm trying to set up a snapshot repository to get a consistent backup (in order to use elasticsearch as primary data store).

When I try to create the repository, I get an error 500:
error: "RemoteTransportException[[Jack Kirby][inet[/10.133.199.7:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[hurz] [_kKn5dODRzmPOhi05U59yg, 'RemoteTransportException[[Box][inet[/10.133.199.63:9300]][internal:admin/repository/verify]]; nested: RepositoryVerificationException[[hurz] store location [/tmp/ess] is not shared between node [[Box][_kKn5dODRzmPOhi05U59yg][esstest-a][inet[/10.133.199.63:9300]]] and the master node]; ']]]; "

I can see with inotify that it creates two files like
tests-yOvEwIjQQP6rs_xqf-iFAQ-master
tests-yOvEwIjQQP6rs_xqf-iFAQ-WKG7i2KATZWalcDgwEfX8Q

But those two files are removed immediately after, before they even arrive on the other node.

How long does elasticsearch wait for the files to appear on the other node before it decides that the store location "is not shared"?

In general I think having to have a shared filesystem for backup takes away some of the advantage of elasticsearch replicating itself. If I have a shared filesystem anyway, I could just put the index there. For indexes that are not continuously updated of course.

You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/02af3bdc-f5e2-418d-b8c9-2aeeaf245a4a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e44737d6-8fdf-486f-883c-33435e8b9cd4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4d274fc6-7612-4482-8264-6293c3ad6b02%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f6a9f812-6656-46e0-ab5f-f212671b29d1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/36BBB027-811E-44BD-A9C5-36F9461F468B%40pilato.fr.
For more options, visit https://groups.google.com/d/optout.