Just Pushed: Local Gateway

Elastic Stack Elasticsearch
1

Hi,

I just pushed support for a `local` gateway:

http://github.com/elasticsearch/elasticsearch/issues/issue/343. The idea of
the local gateway is to allow to perform full cluster recovery from local
information each node stores and not require a shared storage gateway (like
fs for shared file system, or s3, and hdfs).

The idea of a shared gateway works really well when local node data is

considered transient, but a full cluster recovery is still required. A
transient local data can be when deciding to store the index in memory.
Another benefit of using a shared storage gateway is to allow for easy
backup of the index. If a backup process is required (on top of the high
availability aspect of elasticsearch), it make sense to have it integrated
into how elasticsearch works, so the recovery process will take the backups
into account.

Still, there are many cases where reusing the local data stored on each

node when performing full recovery make sense. Very large indices for
example, that are good with N (>1) replicas per shard, might find the shared
storage an overhead. Note that the shared storage model is not something
that other nosql solution provide, and most people are ok with relying on
local node storage and increased number of replicas. The good news is that
elasticsearch provides it now.

As a side note, shared storage model is something that takes some time

for people to understand (I can't count the number of times I have heard
something like: "ahh, it requires a shared storage, its not really
distributed then..."). Its a given when talking about "in memory" data
grids, since when bringing down the whole cluster, data was lost, and there
should be a way to recover it. With elasticsearch, this was my first
architecture decision for long term persistency, but I always had on the
roadmap the mentioned support.

Of course, this only make sense when using file based index storage (the

transaction log has moved to be file based since 0.8).

Enabling it is quite simple:

gateway:
type: "local"

Usually, you would want to configure the "gateway.recover_after_nodes"

setting to allow for more nodes to be in play when performing full cluster
restart so the correct cluster state is elected.

Last, I am considering make this setting the default one that comes with

the elasticsearch.yml setting. What do you think?

-shay.banon

2

Hey Shay,
Cool stuff. Makes sense to me to be the default, but what ends up
happening when a shard that is needed isn't on any of the local boxes
after a server restart?

Thanks,
Paul

On Aug 31, 1:26 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

I just pushed support for a `local` gateway:http://github.com/elasticsearch/elasticsearch/issues/issue/343. The idea of

the local gateway is to allow to perform full cluster recovery from local
information each node stores and not require a shared storage gateway (like
fs for shared file system, or s3, and hdfs).

The idea of a shared gateway works really well when local node data is

considered transient, but a full cluster recovery is still required. A
transient local data can be when deciding to store the index in memory.
Another benefit of using a shared storage gateway is to allow for easy
backup of the index. If a backup process is required (on top of the high
availability aspect of elasticsearch), it make sense to have it integrated
into how elasticsearch works, so the recovery process will take the backups
into account.

Still, there are many cases where reusing the local data stored on each

node when performing full recovery make sense. Very large indices for
example, that are good with N (>1) replicas per shard, might find the shared
storage an overhead. Note that the shared storage model is not something
that other nosql solution provide, and most people are ok with relying on
local node storage and increased number of replicas. The good news is that
elasticsearch provides it now.

As a side note, shared storage model is something that takes some time

for people to understand (I can't count the number of times I have heard
something like: "ahh, it requires a shared storage, its not really
distributed then..."). Its a given when talking about "in memory" data
grids, since when bringing down the whole cluster, data was lost, and there
should be a way to recover it. With elasticsearch, this was my first
architecture decision for long term persistency, but I always had on the
roadmap the mentioned support.

Of course, this only make sense when using file based index storage (the

transaction log has moved to be file based since 0.8).

Enabling it is quite simple:

gateway:
type: "local"

Usually, you would want to configure the "gateway.recover_after_nodes"

setting to allow for more nodes to be in play when performing full cluster
restart so the correct cluster state is elected.

Last, I am considering make this setting the default one that comes with

the elasticsearch.yml setting. What do you think?

-shay.banon

3

Hi Kimchy

Usually, you would want to configure the

"gateway.recover_after_nodes" setting to allow for more nodes to be in
play when performing full cluster restart so the correct cluster state
is elected.

So what would happen if two nodes have different versions of the same
shard stored locally? I presume it'd take the more recent version. But
what it if has already restarted and used the older version?

What about situations when the cluster has split, so each node has a
version of the shard with changes that the other node doesn't have?

Is there any (efficient) way to merge these?

Last, I am considering make this setting the default one that

comes with the elasticsearch.yml setting. What do you think?

It would certainly make getting started simpler for the newly arrived.
Instant persistent storage without configuration.

clint

4

Hi,

While not all shards are found on the currently discovered nodes, the
index is in a "blocked" state and operations are not allowed on it. Note
that replicas are in this game as well.

The versioning is separated into two. The first is trying to find the
latest "meta data", which include the indices created, their settings and
mapping. This will happen once the "recover_after_nodes" nodes are found,
and the latest version is chosen.

Then there is the part where for each index, once all the relevant shards
are found for it, then they will be used. The "latest" shard versions will
be used and allocated.

If there is a split the involves more than one node (i.e. you have a 20

node cluster that got split in half), the way that it works now is that both
will become separated clusters, with their own versions and so on. If a full
cluster restart is done, and all 20 nodes are back into the picture, then
the one that managed to get to a higher version will win.

-shay.banon

On Tue, Aug 31, 2010 at 10:38 PM, Clinton Gormley
clinton@iannounce.co.ukwrote:

Hi Kimchy

Usually, you would want to configure the

"gateway.recover_after_nodes" setting to allow for more nodes to be in
play when performing full cluster restart so the correct cluster state
is elected.

So what would happen if two nodes have different versions of the same
shard stored locally? I presume it'd take the more recent version. But
what it if has already restarted and used the older version?

What about situations when the cluster has split, so each node has a
version of the shard with changes that the other node doesn't have?

Is there any (efficient) way to merge these?

Last, I am considering make this setting the default one that

comes with the elasticsearch.yml setting. What do you think?

It would certainly make getting started simpler for the newly arrived.
Instant persistent storage without configuration.

clint

5

+1 for making local storage the default config. Although I think most
production implementations will use shared storage, most people start with a
single instance, and shared storage is not familiar to many. Making local
storage would eliminate one more thing to learn in the beginning.
As time goes, people can see the pros/cons of shared storage themselves and
choose the right config.

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Tue, Aug 31, 2010 at 3:26 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi,

I just pushed support for a `local` gateway:

Gateway: a `local` gateway · Issue #343 · elastic/elasticsearch · GitHub. The idea
of the local gateway is to allow to perform full cluster recovery from local
information each node stores and not require a shared storage gateway (like
fs for shared file system, or s3, and hdfs).

The idea of a shared gateway works really well when local node data is

considered transient, but a full cluster recovery is still required. A
transient local data can be when deciding to store the index in memory.
Another benefit of using a shared storage gateway is to allow for easy
backup of the index. If a backup process is required (on top of the high
availability aspect of elasticsearch), it make sense to have it integrated
into how elasticsearch works, so the recovery process will take the backups
into account.

Still, there are many cases where reusing the local data stored on each

node when performing full recovery make sense. Very large indices for
example, that are good with N (>1) replicas per shard, might find the shared
storage an overhead. Note that the shared storage model is not something
that other nosql solution provide, and most people are ok with relying on
local node storage and increased number of replicas. The good news is that
elasticsearch provides it now.

As a side note, shared storage model is something that takes some time

for people to understand (I can't count the number of times I have heard
something like: "ahh, it requires a shared storage, its not really
distributed then..."). Its a given when talking about "in memory" data
grids, since when bringing down the whole cluster, data was lost, and there
should be a way to recover it. With elasticsearch, this was my first
architecture decision for long term persistency, but I always had on the
roadmap the mentioned support.

Of course, this only make sense when using file based index storage

(the transaction log has moved to be file based since 0.8).

Enabling it is quite simple:

gateway:
type: "local"

Usually, you would want to configure the "gateway.recover_after_nodes"

setting to allow for more nodes to be in play when performing full cluster
restart so the correct cluster state is elected.

Last, I am considering make this setting the default one that comes

with the elasticsearch.yml setting. What do you think?

-shay.banon

6

Yes, agreed. Just want to stress the point that "local" gateway will also
work in distributed environment, and actually, I can see it being used at
the way to support long term persistency for different type of applications
(like I noted on my first email).

-shay.banon

On Tue, Aug 31, 2010 at 11:57 PM, Berkay Mollamustafaoglu <mberkay@gmail.com

wrote:

+1 for making local storage the default config. Although I think most
production implementations will use shared storage, most people start with a
single instance, and shared storage is not familiar to many. Making local
storage would eliminate one more thing to learn in the beginning.
As time goes, people can see the pros/cons of shared storage themselves and
choose the right config.

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Tue, Aug 31, 2010 at 3:26 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi,

I just pushed support for a `local` gateway:

Gateway: a `local` gateway · Issue #343 · elastic/elasticsearch · GitHub. The idea
of the local gateway is to allow to perform full cluster recovery from local
information each node stores and not require a shared storage gateway (like
fs for shared file system, or s3, and hdfs).

The idea of a shared gateway works really well when local node data is

considered transient, but a full cluster recovery is still required. A
transient local data can be when deciding to store the index in memory.
Another benefit of using a shared storage gateway is to allow for easy
backup of the index. If a backup process is required (on top of the high
availability aspect of elasticsearch), it make sense to have it integrated
into how elasticsearch works, so the recovery process will take the backups
into account.

Still, there are many cases where reusing the local data stored on

each node when performing full recovery make sense. Very large indices for
example, that are good with N (>1) replicas per shard, might find the shared
storage an overhead. Note that the shared storage model is not something
that other nosql solution provide, and most people are ok with relying on
local node storage and increased number of replicas. The good news is that
elasticsearch provides it now.

As a side note, shared storage model is something that takes some time

for people to understand (I can't count the number of times I have heard
something like: "ahh, it requires a shared storage, its not really
distributed then..."). Its a given when talking about "in memory" data
grids, since when bringing down the whole cluster, data was lost, and there
should be a way to recover it. With elasticsearch, this was my first
architecture decision for long term persistency, but I always had on the
roadmap the mentioned support.

Of course, this only make sense when using file based index storage

(the transaction log has moved to be file based since 0.8).

Enabling it is quite simple:

gateway:
type: "local"

Usually, you would want to configure the "gateway.recover_after_nodes"

setting to allow for more nodes to be in play when performing full cluster
restart so the correct cluster state is elected.

Last, I am considering make this setting the default one that comes

with the elasticsearch.yml setting. What do you think?

-shay.banon

7

Is there any other configuration to be done for the local gateway
type, such as filesystem location, etc?

-Andrei

On Aug 31, 12:26 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

I just pushed support for a `local` gateway:http://github.com/elasticsearch/elasticsearch/issues/issue/343. The idea of

the local gateway is to allow to perform full cluster recovery from local
information each node stores and not require a shared storage gateway (like
fs for shared file system, or s3, and hdfs).

The idea of a shared gateway works really well when local node data is

considered transient, but a full cluster recovery is still required. A
transient local data can be when deciding to store the index in memory.
Another benefit of using a shared storage gateway is to allow for easy
backup of the index. If a backup process is required (on top of the high
availability aspect of elasticsearch), it make sense to have it integrated
into how elasticsearch works, so the recovery process will take the backups
into account.

Still, there are many cases where reusing the local data stored on each

node when performing full recovery make sense. Very large indices for
example, that are good with N (>1) replicas per shard, might find the shared
storage an overhead. Note that the shared storage model is not something
that other nosql solution provide, and most people are ok with relying on
local node storage and increased number of replicas. The good news is that
elasticsearch provides it now.

As a side note, shared storage model is something that takes some time

for people to understand (I can't count the number of times I have heard
something like: "ahh, it requires a shared storage, its not really
distributed then..."). Its a given when talking about "in memory" data
grids, since when bringing down the whole cluster, data was lost, and there
should be a way to recover it. With elasticsearch, this was my first
architecture decision for long term persistency, but I always had on the
roadmap the mentioned support.

Of course, this only make sense when using file based index storage (the

transaction log has moved to be file based since 0.8).

Enabling it is quite simple:

gateway:
type: "local"

Usually, you would want to configure the "gateway.recover_after_nodes"

setting to allow for more nodes to be in play when performing full cluster
restart so the correct cluster state is elected.

Last, I am considering make this setting the default one that comes with

the elasticsearch.yml setting. What do you think?

-shay.banon

8

It defaults to the work location. If you want to change it, then you can
change the work location.

On Thu, Sep 30, 2010 at 7:12 AM, Andrei andrei@zmievski.org wrote:

Is there any other configuration to be done for the local gateway
type, such as filesystem location, etc?

-Andrei

On Aug 31, 12:26 pm, Shay Banon shay.ba...@elasticsearch.com wrote:

Hi,

I just pushed support for a `local` gateway:

Gateway: a `local` gateway · Issue #343 · elastic/elasticsearch · GitHub. The idea
of

the local gateway is to allow to perform full cluster recovery from local
information each node stores and not require a shared storage gateway
(like
fs for shared file system, or s3, and hdfs).

The idea of a shared gateway works really well when local node data

is

considered transient, but a full cluster recovery is still required. A
transient local data can be when deciding to store the index in memory.
Another benefit of using a shared storage gateway is to allow for easy
backup of the index. If a backup process is required (on top of the high
availability aspect of elasticsearch), it make sense to have it
integrated
into how elasticsearch works, so the recovery process will take the
backups
into account.

Still, there are many cases where reusing the local data stored on

each

node when performing full recovery make sense. Very large indices for
example, that are good with N (>1) replicas per shard, might find the
shared
storage an overhead. Note that the shared storage model is not something
that other nosql solution provide, and most people are ok with relying on
local node storage and increased number of replicas. The good news is
that
elasticsearch provides it now.

As a side note, shared storage model is something that takes some

time

for people to understand (I can't count the number of times I have heard
something like: "ahh, it requires a shared storage, its not really
distributed then..."). Its a given when talking about "in memory" data
grids, since when bringing down the whole cluster, data was lost, and
there
should be a way to recover it. With elasticsearch, this was my first
architecture decision for long term persistency, but I always had on the
roadmap the mentioned support.

Of course, this only make sense when using file based index storage

(the

transaction log has moved to be file based since 0.8).

Enabling it is quite simple:

gateway:
type: "local"

Usually, you would want to configure the

"gateway.recover_after_nodes"

setting to allow for more nodes to be in play when performing full
cluster
restart so the correct cluster state is elected.

Last, I am considering make this setting the default one that comes

with

the elasticsearch.yml setting. What do you think?

-shay.banon