I've been doing research against various distributed search systems as
well as "NoSQL" K/V storage systems and have come to look into
utilizing ElasticSearch for our searching (and possibly for our K/V as
well). However, I have a few questions regarding it's durability and
distributed nature since I can't seem to find much information online
about this.
I've seen mention of a "master node" that maintains global state and
reassigns shards. How does the system react if the "master"
partitions from the cluster for any reason (network partition, hdd
crash, etc)? Does another node self-promote? What happens when this
"master" comes back online?
I've also seen that shards and backups are utlized in ElasticSearch
instead of a "ring" as defined in the Dynamo paper. This sounds
similar to other distributed databases such as MongoDB (shards,
replica sets, and replicas) but I can't seem to find any information
about this. Is there somewhere online that I can review the sharding
and replica information? I reviewed
http://www.elasticsearch.org/blog/2010/02/16/searchengine_time_machine.html
but it doesn't quite explain everything. I'm looking to see how an
index is broken into shards, what happens when shard replicas are
partitioned from the group at the time of a write (do they "sync" up
when they come back online), and even how the long term peristency
with the local gateway storage, and what happens when we crash before
information is written to this long term storage?
Any information is appreciated,
Jason