Value of Gateway in non-cloud environments


(Otis Gospodnetić) #1

Hi,

I've read various information about the concept of Gateway in ES and
I'm wondering what value Gateway adds when one is not deploying ES in
a cloud? In a cloud like AWS local storage is not necessarily
persistent - when you stop an EC2 instance I believe its local data is
gone. But what if you are deploying ES to a set of your own servers
where this is not the case? Which type of gw should one use? local?
fs?

Thanks,
Otis

Sematext is hiring search engineers - http://sematext.com/about/jobs.html


(Shay Banon) #2

local, the default, is the recommended one, and actually it is also the one
being used in most cases in cloud environments as well.

On Fri, Jul 29, 2011 at 2:03 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

I've read various information about the concept of Gateway in ES and
I'm wondering what value Gateway adds when one is not deploying ES in
a cloud? In a cloud like AWS local storage is not necessarily
persistent - when you stop an EC2 instance I believe its local data is
gone. But what if you are deploying ES to a set of your own servers
where this is not the case? Which type of gw should one use? local?
fs?

Thanks,
Otis

Sematext is hiring search engineers - http://sematext.com/about/jobs.html


(Otis Gospodnetić) #3

Thanks Shay.

In case of "local" all index data stays local spread over N replicas
and M shards.
And the index settings/configs/mappings/state/etc. is all stored
locally (on local disks), right? And what about replicas of this
data? Is this stored in 1 central place or are there copies/replicas
of index settings/configs/mappings/state/etc. on multiple cluster
nodes? I couldn't find this stated in the guide or anywhere else.

Thanks,
Otis

Sematext is hiring search engineers - http://sematext.com/about/jobs.html

On Jul 29, 2:19 am, Shay Banon kim...@gmail.com wrote:

local, the default, is the recommended one, and actually it is also the one
being used in most cases in cloud environments as well.

On Fri, Jul 29, 2011 at 2:03 AM, Otis Gospodnetic <

otis.gospodne...@gmail.com> wrote:

Hi,

I've read various information about the concept of Gateway in ES and
I'm wondering what value Gateway adds when one is not deploying ES in
a cloud? In a cloud like AWS local storage is not necessarily
persistent - when you stop an EC2 instance I believe its local data is
gone. But what if you are deploying ES to a set of your own servers
where this is not the case? Which type of gw should one use? local?
fs?

Thanks,
Otis

Sematext is hiring search engineers -http://sematext.com/about/jobs.html


(Shay Banon) #4

Thats stored on all the eligible master nodes (which in a default config,
its all nodes).

On Fri, Jul 29, 2011 at 11:35 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Thanks Shay.

In case of "local" all index data stays local spread over N replicas
and M shards.
And the index settings/configs/mappings/state/etc. is all stored
locally (on local disks), right? And what about replicas of this
data? Is this stored in 1 central place or are there copies/replicas
of index settings/configs/mappings/state/etc. on multiple cluster
nodes? I couldn't find this stated in the guide or anywhere else.

Thanks,
Otis

Sematext is hiring search engineers - http://sematext.com/about/jobs.html

On Jul 29, 2:19 am, Shay Banon kim...@gmail.com wrote:

local, the default, is the recommended one, and actually it is also the
one
being used in most cases in cloud environments as well.

On Fri, Jul 29, 2011 at 2:03 AM, Otis Gospodnetic <

otis.gospodne...@gmail.com> wrote:

Hi,

I've read various information about the concept of Gateway in ES and
I'm wondering what value Gateway adds when one is not deploying ES in
a cloud? In a cloud like AWS local storage is not necessarily
persistent - when you stop an EC2 instance I believe its local data is
gone. But what if you are deploying ES to a set of your own servers
where this is not the case? Which type of gw should one use? local?
fs?

Thanks,
Otis

Sematext is hiring search engineers -
http://sematext.com/about/jobs.html


(Otis Gospodnetić) #5

Hi,

Would it then be true to say that pretty much the only (or at least
the main) scenario where Shared FS gw makes more sense over the Local
gw is when one is dealing with servers whose local storage is
ephemeral/temporary, as is the case in EC2?

Thanks,
Otis

Sematext is hiring search engineers - http://sematext.com/about/jobs.html

On Jul 29, 6:33 am, Shay Banon kim...@gmail.com wrote:

Thats stored on all the eligible master nodes (which in a default config,
its all nodes).

On Fri, Jul 29, 2011 at 11:35 AM, Otis Gospodnetic <

otis.gospodne...@gmail.com> wrote:

Thanks Shay.

In case of "local" all index data stays local spread over N replicas
and M shards.
And the index settings/configs/mappings/state/etc. is all stored
locally (on local disks), right? And what about replicas of this
data? Is this stored in 1 central place or are there copies/replicas
of index settings/configs/mappings/state/etc. on multiple cluster
nodes? I couldn't find this stated in the guide or anywhere else.

Thanks,
Otis

Sematext is hiring search engineers -http://sematext.com/about/jobs.html

On Jul 29, 2:19 am, Shay Banon kim...@gmail.com wrote:

local, the default, is the recommended one, and actually it is also the
one
being used in most cases in cloud environments as well.

On Fri, Jul 29, 2011 at 2:03 AM, Otis Gospodnetic <

otis.gospodne...@gmail.com> wrote:

Hi,

I've read various information about the concept of Gateway in ES and
I'm wondering what value Gateway adds when one is not deploying ES in
a cloud? In a cloud like AWS local storage is not necessarily
persistent - when you stop an EC2 instance I believe its local data is
gone. But what if you are deploying ES to a set of your own servers
where this is not the case? Which type of gw should one use? local?
fs?

Thanks,
Otis

Sematext is hiring search engineers -
http://sematext.com/about/jobs.html


(Shay Banon) #6

Yes, that is the most common case. Another use case is for cases where an in
memory index is used, but long term persistency is still desirable in a
write behind fashion.

Note that even with the ec2 and local storage its not a clear cut. You can
still use local gateway with a high replica account, if you are ok with the
fact that if (number_of_replicas + 1) nodes are wiped out, you need to
reindex the data.

-shay.banon

On Fri, Jul 29, 2011 at 11:10 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

Hi,

Would it then be true to say that pretty much the only (or at least
the main) scenario where Shared FS gw makes more sense over the Local
gw is when one is dealing with servers whose local storage is
ephemeral/temporary, as is the case in EC2?

Thanks,
Otis

Sematext is hiring search engineers - http://sematext.com/about/jobs.html

On Jul 29, 6:33 am, Shay Banon kim...@gmail.com wrote:

Thats stored on all the eligible master nodes (which in a default config,
its all nodes).

On Fri, Jul 29, 2011 at 11:35 AM, Otis Gospodnetic <

otis.gospodne...@gmail.com> wrote:

Thanks Shay.

In case of "local" all index data stays local spread over N replicas
and M shards.
And the index settings/configs/mappings/state/etc. is all stored
locally (on local disks), right? And what about replicas of this
data? Is this stored in 1 central place or are there copies/replicas
of index settings/configs/mappings/state/etc. on multiple cluster
nodes? I couldn't find this stated in the guide or anywhere else.

Thanks,
Otis

Sematext is hiring search engineers -
http://sematext.com/about/jobs.html

On Jul 29, 2:19 am, Shay Banon kim...@gmail.com wrote:

local, the default, is the recommended one, and actually it is also
the

one

being used in most cases in cloud environments as well.

On Fri, Jul 29, 2011 at 2:03 AM, Otis Gospodnetic <

otis.gospodne...@gmail.com> wrote:

Hi,

I've read various information about the concept of Gateway in ES
and

I'm wondering what value Gateway adds when one is not deploying ES
in

a cloud? In a cloud like AWS local storage is not necessarily
persistent - when you stop an EC2 instance I believe its local data
is

gone. But what if you are deploying ES to a set of your own
servers

where this is not the case? Which type of gw should one use?
local?

fs?

Thanks,
Otis

Sematext is hiring search engineers -
http://sematext.com/about/jobs.html


(system) #7