Elasticsearch hadoop plugin

Alexis_Okuwa · March 3, 2013, 12:13am

I was thinking about using the hadoop plugin for a project and saw that it
is getting depercated but the project seems to be active on github, is
there a reason its getting depercated

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · March 4, 2013, 9:43am

The reason is, shared gateways do not scale well enough (local is much
better) and stand in the way for implementing a new smart backup/restore
mechanism. In fact, most people intend backups on HDFS/S3/NFS, but
setting a shared gateway is just a snapshot of the index shared by many
nodes, not a scalable backup/restore solution.

github.com/elastic/elasticsearch

Deprecate Shared Gateway

opened 10:43AM - 03 Dec 12 UTC

closed 10:44AM - 03 Dec 12 UTC

kimchy

>breaking v0.20.0 v0.90.0.Beta1 v0.19.12

Shared gateways (shared FS storage or S3 for example) are problematic performanc…e wise since they constantly need to snapshot the state of the index to a shared location, and then use that as the system of record. The local gateway on the other hand doesn't need it, and performs much better. The main benefit of a shared gateway is the fact that the data is actually stored on another persistent location (i.e. using ephemeral disks on AWS, but still having the data on s3), but then its actually abusing the shared gateway design (to be used as a backup). In the near future, we will have a proper snapshot(backup)/restore API, which will be the proper way to do backups, but relaying on the shared gateway for that is problematic. Note, backups can still be made by "rsync" the data location for each node "manually".

I think, after backup/restore is added to ES, you will be able to use
HDFS again.

Jörg

Am 03.03.2013 01:13, schrieb Wojons Tech:

I was thinking about using the hadoop plugin for a project and saw
that it is getting depercated but the project seems to be active on
github, is there a reason its getting depercated --
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.