Good candidate for a gateway?


(Lukáš Vlček) #1

Hi,

what kind of features must have the system that is good candidate for the
gateway storage? With all those distributed and replicated key value stores
these days how can I learn that specific product is good for ES gateway? As
far as I understand the traffic is not critical (ES stores into gateway
asynchronously and not "very often"), however, ability to store blobs is
critical (system which can handle a lot of small key-value entities but can
not handle large binary entities is probably not good candidate). Once data
is stored in gateway it should be reliable (system that can not cope with
crashes is not a good candidate). Are there any other requirements?

Regards,
Lukas


(Shay Banon) #2

Basically, this is it. Note that key value stores like cassandra also make
sense since you can always represent a blob as chunks of bytes. Actually,
since this is so common scenario, in elasticsearch there is built in support
for chunking blobs.

-shay.banon

On Sat, Jul 31, 2010 at 10:58 PM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi,

what kind of features must have the system that is good candidate for the
gateway storage? With all those distributed and replicated key value stores
these days how can I learn that specific product is good for ES gateway? As
far as I understand the traffic is not critical (ES stores into gateway
asynchronously and not "very often"), however, ability to store blobs is
critical (system which can handle a lot of small key-value entities but can
not handle large binary entities is probably not good candidate). Once data
is stored in gateway it should be reliable (system that can not cope with
crashes is not a good candidate). Are there any other requirements?

Regards,
Lukas


(Lukáš Vlček) #3

What would be the best way if I want to evaluate some key-value store or
distributed cache if it plays nice as a ES gateway? Is there any simple
gateway interface I have to implement and any other ES API I need to
consider when implementing it?

Lukas

On Sun, Aug 1, 2010 at 1:10 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Basically, this is it. Note that key value stores like cassandra also make
sense since you can always represent a blob as chunks of bytes. Actually,
since this is so common scenario, in elasticsearch there is built in support
for chunking blobs.

-shay.banon

On Sat, Jul 31, 2010 at 10:58 PM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Hi,

what kind of features must have the system that is good candidate for the
gateway storage? With all those distributed and replicated key value stores
these days how can I learn that specific product is good for ES gateway? As
far as I understand the traffic is not critical (ES stores into gateway
asynchronously and not "very often"), however, ability to store blobs is
critical (system which can handle a lot of small key-value entities but can
not handle large binary entities is probably not good candidate). Once data
is stored in gateway it should be reliable (system that can not cope with
crashes is not a good candidate). Are there any other requirements?

Regards,
Lukas


(Shay Banon) #4

You can have a look at the Hadoop and S3 gateways. There is a common
abstraction on top of blob store (very simple) and a built in base classes
for gateway that uses this blob store.

-shay.banon

On Sun, Aug 1, 2010 at 2:19 PM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

What would be the best way if I want to evaluate some key-value store or
distributed cache if it plays nice as a ES gateway? Is there any simple
gateway interface I have to implement and any other ES API I need to
consider when implementing it?

Lukas

On Sun, Aug 1, 2010 at 1:10 PM, Shay Banon shay.banon@elasticsearch.comwrote:

Basically, this is it. Note that key value stores like cassandra also make
sense since you can always represent a blob as chunks of bytes. Actually,
since this is so common scenario, in elasticsearch there is built in support
for chunking blobs.

-shay.banon

On Sat, Jul 31, 2010 at 10:58 PM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Hi,

what kind of features must have the system that is good candidate for the
gateway storage? With all those distributed and replicated key value stores
these days how can I learn that specific product is good for ES gateway? As
far as I understand the traffic is not critical (ES stores into gateway
asynchronously and not "very often"), however, ability to store blobs is
critical (system which can handle a lot of small key-value entities but can
not handle large binary entities is probably not good candidate). Once data
is stored in gateway it should be reliable (system that can not cope with
crashes is not a good candidate). Are there any other requirements?

Regards,
Lukas


(system) #5