Shared File System Gateway behavior


(sujoysett) #1

Hi,

I am indexing documents from social media into an ES index spanned over two
nodes (5 shards 1 replica) on two physical machines, and I am using a third
machine as location for the Shared FileSystem Gateway. What happens when
this third machine hosting the gateway gets corrupted for any reason? Let
us assume a situation, all three machines are restarted, and somehow
the third machine hosting the gateway is not starting.

In such case, ES fails to start stating that the Gateway Location is
unavailable. If we define some other location (empty folder) as gateway, ES
starts, but it does not find the metadata of indexes, shows a message that
local indexes are dangling indexes, and schedules to delete them from the
local store.

In other words, if a shared gateway location is permanently lost, what is
the procedure to make a copy again from local store to a new gateway
location?

Regards,
Sujoy.


(Shay Banon) #2

If the shared gateway data is lost, then you can't recover from it. Use
local gateway, it is the preferred solution.

On Fri, May 18, 2012 at 8:40 AM, Sujoy Sett sujoysett@gmail.com wrote:

Hi,

I am indexing documents from social media into an ES index spanned over
two nodes (5 shards 1 replica) on two physical machines, and I am using a
third machine as location for the Shared FileSystem Gateway. What happens
when this third machine hosting the gateway gets corrupted for any reason?
Let us assume a situation, all three machines are restarted, and somehow
the third machine hosting the gateway is not starting.

In such case, ES fails to start stating that the Gateway Location is
unavailable. If we define some other location (empty folder) as gateway, ES
starts, but it does not find the metadata of indexes, shows a message that
local indexes are dangling indexes, and schedules to delete them from the
local store.

In other words, if a shared gateway location is permanently lost, what is
the procedure to make a copy again from local store to a new gateway
location?

Regards,
Sujoy.


(sujoysett) #3

Thanks Shay.

I will consider moving back to local gateway in that case.

But is there any chance of getting API control of moving data from local
copy to gateway, instead of the normal flow, if required, for such cases as
I mentioned above, in future?

And any suggestion on backup strategy? This shared gateway was being used
as single point of weekly disk backup for a cluster of two nodes. If I move
to local gateway, backup would have to be done in the local servers
separately, which becomes increasingly complex when more nodes are involved.

Regards,
Sujoy.

On Monday, May 21, 2012 1:48:44 AM UTC+5:30, kimchy wrote:

If the shared gateway data is lost, then you can't recover from it. Use
local gateway, it is the preferred solution.

On Fri, May 18, 2012 at 8:40 AM, Sujoy Sett wrote:

Hi,

I am indexing documents from social media into an ES index spanned over
two nodes (5 shards 1 replica) on two physical machines, and I am using a
third machine as location for the Shared FileSystem Gateway. What happens
when this third machine hosting the gateway gets corrupted for any reason?
Let us assume a situation, all three machines are restarted, and somehow
the third machine hosting the gateway is not starting.

In such case, ES fails to start stating that the Gateway Location is
unavailable. If we define some other location (empty folder) as gateway, ES
starts, but it does not find the metadata of indexes, shows a message that
local indexes are dangling indexes, and schedules to delete them from the
local store.

In other words, if a shared gateway location is permanently lost, what is
the procedure to make a copy again from local store to a new gateway
location?

Regards,
Sujoy.


(Shay Banon) #4

Hi,

Yes, one thing that is missing is a backup/restore API when using the
local gateway. For now, you can simply backup the data location of each
node.

On Mon, May 21, 2012 at 9:31 AM, Sujoy Sett sujoysett@gmail.com wrote:

Thanks Shay.

I will consider moving back to local gateway in that case.

But is there any chance of getting API control of moving data from local
copy to gateway, instead of the normal flow, if required, for such cases as
I mentioned above, in future?

And any suggestion on backup strategy? This shared gateway was being used
as single point of weekly disk backup for a cluster of two nodes. If I move
to local gateway, backup would have to be done in the local servers
separately, which becomes increasingly complex when more nodes are involved.

Regards,
Sujoy.

On Monday, May 21, 2012 1:48:44 AM UTC+5:30, kimchy wrote:

If the shared gateway data is lost, then you can't recover from it. Use
local gateway, it is the preferred solution.

On Fri, May 18, 2012 at 8:40 AM, Sujoy Sett wrote:

Hi,

I am indexing documents from social media into an ES index spanned over
two nodes (5 shards 1 replica) on two physical machines, and I am using a
third machine as location for the Shared FileSystem Gateway. What happens
when this third machine hosting the gateway gets corrupted for any reason?
Let us assume a situation, all three machines are restarted, and somehow
the third machine hosting the gateway is not starting.

In such case, ES fails to start stating that the Gateway Location is
unavailable. If we define some other location (empty folder) as gateway, ES
starts, but it does not find the metadata of indexes, shows a message that
local indexes are dangling indexes, and schedules to delete them from the
local store.

In other words, if a shared gateway location is permanently lost, what
is the procedure to make a copy again from local store to a new gateway
location?

Regards,
Sujoy.


(sujoysett) #5

Thanks Shay. We will be doing that.

Regards,
Sujoy.

On Thursday, May 24, 2012 2:38:58 AM UTC+5:30, kimchy wrote:

Hi,

Yes, one thing that is missing is a backup/restore API when using the
local gateway. For now, you can simply backup the data location of each
node.

On Mon, May 21, 2012 at 9:31 AM, Sujoy Sett wrote:

Thanks Shay.

I will consider moving back to local gateway in that case.

But is there any chance of getting API control of moving data from local
copy to gateway, instead of the normal flow, if required, for such cases as
I mentioned above, in future?

And any suggestion on backup strategy? This shared gateway was being used
as single point of weekly disk backup for a cluster of two nodes. If I move
to local gateway, backup would have to be done in the local servers
separately, which becomes increasingly complex when more nodes are involved.

Regards,
Sujoy.

On Monday, May 21, 2012 1:48:44 AM UTC+5:30, kimchy wrote:

If the shared gateway data is lost, then you can't recover from it. Use
local gateway, it is the preferred solution.

On Fri, May 18, 2012 at 8:40 AM, Sujoy Sett wrote:

Hi,

I am indexing documents from social media into an ES index spanned over
two nodes (5 shards 1 replica) on two physical machines, and I am using a
third machine as location for the Shared FileSystem Gateway. What happens
when this third machine hosting the gateway gets corrupted for any reason?
Let us assume a situation, all three machines are restarted, and somehow
the third machine hosting the gateway is not starting.

In such case, ES fails to start stating that the Gateway Location is
unavailable. If we define some other location (empty folder) as gateway, ES
starts, but it does not find the metadata of indexes, shows a message that
local indexes are dangling indexes, and schedules to delete them from the
local store.

In other words, if a shared gateway location is permanently lost, what
is the procedure to make a copy again from local store to a new gateway
location?

Regards,
Sujoy.


(system) #6