Announcement: ZooKeeper Discovery Plugin for Elasticsearch


(Igor Motov) #1

Sonian, Inc. has released ZooKeeper Discovery Plugin as open source under
Apache 2.0 License. The ZooKeeper plugin is a drop-in replacement for Zen
Discovery. It uses Apache ZooKeeper for master election, node fault
detection and state publishing. This plugin is still experimental and not
recommended to be used in production environments.

For more information visit plugin project page:


(Otis Gospodnetić) #2

Very nice.

When and/or why should one use this instead of Zen Discovery? What are
pros/cons of one vs. the other?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Tuesday, July 10, 2012 8:14:12 PM UTC-4, Igor Motov wrote:

Sonian, Inc. has released ZooKeeper Discovery Plugin as open source under
Apache 2.0 License. The ZooKeeper plugin is a drop-in replacement for Zen
Discovery. It uses Apache ZooKeeper for master election, node fault
detection and state publishing. This plugin is still experimental and not
recommended to be used in production environments.

For more information visit plugin project page:
https://github.com/sonian/elasticsearch-zookeeper


(Igor Motov) #3

ZooKeeper Discovery can simplify setup and configuration in
multicast-unfriendly, dynamically changing environments that are already
using Apache ZooKeeper. ZooKeeper Discovery relies on ZooKeeper to keep
track of nodes in the cluster and perform master election, providing
consistent view of the cluster and identify of the master node. On the
other side, it's not as field tested as Zen Discovery is, so it safe to
assume that it still has some issues to discover and fix.

Igor

On Wednesday, July 11, 2012 12:39:30 AM UTC-4, Otis Gospodnetic wrote:

Very nice.

When and/or why should one use this instead of Zen Discovery? What are
pros/cons of one vs. the other?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Tuesday, July 10, 2012 8:14:12 PM UTC-4, Igor Motov wrote:

Sonian, Inc. has released ZooKeeper Discovery Plugin as open source under
Apache 2.0 License. The ZooKeeper plugin is a drop-in replacement for Zen
Discovery. It uses Apache ZooKeeper for master election, node fault
detection and state publishing. This plugin is still experimental and not
recommended to be used in production environments.

For more information visit plugin project page:
https://github.com/sonian/elasticsearch-zookeeper


(Otis Gospodnetić) #4

Thanks Igor. You should stick that in the README.

Also, does using that make any difference when it comes to split brain?

See:

  1. https://groups.google.com/d/msg/elasticsearch/_QWKLnpvmeU/csbvkqaFF_0J
  2. http://search-lucene.com/m/aN5vk1xuETB1

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, July 11, 2012 11:56:48 AM UTC-4, Igor Motov wrote:

ZooKeeper Discovery can simplify setup and configuration in
multicast-unfriendly, dynamically changing environments that are already
using Apache ZooKeeper. ZooKeeper Discovery relies on ZooKeeper to keep
track of nodes in the cluster and perform master election, providing
consistent view of the cluster and identify of the master node. On the
other side, it's not as field tested as Zen Discovery is, so it safe to
assume that it still has some issues to discover and fix.

Igor

On Wednesday, July 11, 2012 12:39:30 AM UTC-4, Otis Gospodnetic wrote:

Very nice.

When and/or why should one use this instead of Zen Discovery? What are
pros/cons of one vs. the other?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Tuesday, July 10, 2012 8:14:12 PM UTC-4, Igor Motov wrote:

Sonian, Inc. has released ZooKeeper Discovery Plugin as open source
under Apache 2.0 License. The ZooKeeper plugin is a drop-in replacement for
Zen Discovery. It uses Apache ZooKeeper for master election, node fault
detection and state publishing. This plugin is still experimental and not
recommended to be used in production environments.

For more information visit plugin project page:
https://github.com/sonian/elasticsearch-zookeeper


(Igor Motov) #5

Avoiding split brain was one of the original driving forces behind
ZooKeeper Discovery plugin development. Since then elasticsearch split
brain resistance was significantly improved. Indeed, properly
specified discovery.zen.minimum_master_nodes helps clusters to remain
coherent. In case of ZooKeeper Discovery plugin, quorum is maintained by
ZooKeeper nodes, so it's not necessary to specify
discovery.zen.minimum_master_nodes on each elasticsearch node.

Igor

On Wednesday, July 11, 2012 10:59:31 PM UTC-4, Otis Gospodnetic wrote:

Thanks Igor. You should stick that in the README.

Also, does using that make any difference when it comes to split brain?

See:

  1. https://groups.google.com/d/msg/elasticsearch/_QWKLnpvmeU/csbvkqaFF_0J
  2. http://search-lucene.com/m/aN5vk1xuETB1

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, July 11, 2012 11:56:48 AM UTC-4, Igor Motov wrote:

ZooKeeper Discovery can simplify setup and configuration in
multicast-unfriendly, dynamically changing environments that are already
using Apache ZooKeeper. ZooKeeper Discovery relies on ZooKeeper to keep
track of nodes in the cluster and perform master election, providing
consistent view of the cluster and identify of the master node. On the
other side, it's not as field tested as Zen Discovery is, so it safe to
assume that it still has some issues to discover and fix.

Igor

On Wednesday, July 11, 2012 12:39:30 AM UTC-4, Otis Gospodnetic wrote:

Very nice.

When and/or why should one use this instead of Zen Discovery? What are
pros/cons of one vs. the other?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Tuesday, July 10, 2012 8:14:12 PM UTC-4, Igor Motov wrote:

Sonian, Inc. has released ZooKeeper Discovery Plugin as open source
under Apache 2.0 License. The ZooKeeper plugin is a drop-in replacement for
Zen Discovery. It uses Apache ZooKeeper for master election, node fault
detection and state publishing. This plugin is still experimental and not
recommended to be used in production environments.

For more information visit plugin project page:
https://github.com/sonian/elasticsearch-zookeeper


(Otis Gospodnetić) #6

Hi Igor,

Does this mean that the ZK Discovery plugin's only advantage is in
multicast-unfriendly environments? (and I don't mean this in a bad way at
all!) If so, are you planning on some other functionality that would
increase the difference between Zen and ZK Discovery plugin?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Thursday, July 12, 2012 11:38:49 AM UTC-4, Igor Motov wrote:

Avoiding split brain was one of the original driving forces behind
ZooKeeper Discovery plugin development. Since then elasticsearch split
brain resistance was significantly improved. Indeed, properly
specified discovery.zen.minimum_master_nodes helps clusters to remain
coherent. In case of ZooKeeper Discovery plugin, quorum is maintained by
ZooKeeper nodes, so it's not necessary to specify
discovery.zen.minimum_master_nodes on each elasticsearch node.

Igor

On Wednesday, July 11, 2012 10:59:31 PM UTC-4, Otis Gospodnetic wrote:

Thanks Igor. You should stick that in the README.

Also, does using that make any difference when it comes to split brain?

See:

  1. https://groups.google.com/d/msg/elasticsearch/_QWKLnpvmeU/csbvkqaFF_0J
  2. http://search-lucene.com/m/aN5vk1xuETB1

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, July 11, 2012 11:56:48 AM UTC-4, Igor Motov wrote:

ZooKeeper Discovery can simplify setup and configuration in
multicast-unfriendly, dynamically changing environments that are already
using Apache ZooKeeper. ZooKeeper Discovery relies on ZooKeeper to keep
track of nodes in the cluster and perform master election, providing
consistent view of the cluster and identify of the master node. On the
other side, it's not as field tested as Zen Discovery is, so it safe to
assume that it still has some issues to discover and fix.

Igor

On Wednesday, July 11, 2012 12:39:30 AM UTC-4, Otis Gospodnetic wrote:

Very nice.

When and/or why should one use this instead of Zen Discovery? What are
pros/cons of one vs. the other?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Tuesday, July 10, 2012 8:14:12 PM UTC-4, Igor Motov wrote:

Sonian, Inc. has released ZooKeeper Discovery Plugin as open source
under Apache 2.0 License. The ZooKeeper plugin is a drop-in replacement for
Zen Discovery. It uses Apache ZooKeeper for master election, node fault
detection and state publishing. This plugin is still experimental and not
recommended to be used in production environments.

For more information visit plugin project page:
https://github.com/sonian/elasticsearch-zookeeper


(Igor Motov) #7

Hi Otis,

I think I could agree with your statement. ZK Discovery helps avoiding
split-brain in dynamically changing clusters running in
multicast-unfriendly environments without the need to change config files
on servers. There are some things that I would like to improve internally,
but feature-wise it satisfies all our needs at the moment and we don't have
any immediate plans to add any new functionality.

Igor

On Sunday, July 22, 2012 8:02:28 PM UTC-4, Otis Gospodnetic wrote:

Hi Igor,

Does this mean that the ZK Discovery plugin's only advantage is in
multicast-unfriendly environments? (and I don't mean this in a bad way at
all!) If so, are you planning on some other functionality that would
increase the difference between Zen and ZK Discovery plugin?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Thursday, July 12, 2012 11:38:49 AM UTC-4, Igor Motov wrote:

Avoiding split brain was one of the original driving forces behind
ZooKeeper Discovery plugin development. Since then elasticsearch split
brain resistance was significantly improved. Indeed, properly
specified discovery.zen.minimum_master_nodes helps clusters to remain
coherent. In case of ZooKeeper Discovery plugin, quorum is maintained by
ZooKeeper nodes, so it's not necessary to specify
discovery.zen.minimum_master_nodes on each elasticsearch node.

Igor

On Wednesday, July 11, 2012 10:59:31 PM UTC-4, Otis Gospodnetic wrote:

Thanks Igor. You should stick that in the README.

Also, does using that make any difference when it comes to split brain?

See:
1)
https://groups.google.com/d/msg/elasticsearch/_QWKLnpvmeU/csbvkqaFF_0J
2) http://search-lucene.com/m/aN5vk1xuETB1

Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Wednesday, July 11, 2012 11:56:48 AM UTC-4, Igor Motov wrote:

ZooKeeper Discovery can simplify setup and configuration in
multicast-unfriendly, dynamically changing environments that are already
using Apache ZooKeeper. ZooKeeper Discovery relies on ZooKeeper to keep
track of nodes in the cluster and perform master election, providing
consistent view of the cluster and identify of the master node. On the
other side, it's not as field tested as Zen Discovery is, so it safe to
assume that it still has some issues to discover and fix.

Igor

On Wednesday, July 11, 2012 12:39:30 AM UTC-4, Otis Gospodnetic wrote:

Very nice.

When and/or why should one use this instead of Zen Discovery? What
are pros/cons of one vs. the other?

Thanks,
Otis

Search Analytics - http://sematext.com/search-analytics/index.html
Scalable Performance Monitoring - http://sematext.com/spm/index.html

On Tuesday, July 10, 2012 8:14:12 PM UTC-4, Igor Motov wrote:

Sonian, Inc. has released ZooKeeper Discovery Plugin as open source
under Apache 2.0 License. The ZooKeeper plugin is a drop-in replacement for
Zen Discovery. It uses Apache ZooKeeper for master election, node fault
detection and state publishing. This plugin is still experimental and not
recommended to be used in production environments.

For more information visit plugin project page:
https://github.com/sonian/elasticsearch-zookeeper


(Matt Wise) #8

Just wanted to say Thanks for developing this. We were using our own
in-house scripts to coordinate ElasticSearch with ZooKeeper, but your
plugin is excellent. It works very well, and is extremely simple to setup.
I hope you keep developing it for future versions of ZooKeeper.

--Matt
Nextdoor.com, Inc.

On Tuesday, July 10, 2012 5:14:12 PM UTC-7, Igor Motov wrote:

Sonian, Inc. has released ZooKeeper Discovery Plugin as open source under
Apache 2.0 License. The ZooKeeper plugin is a drop-in replacement for Zen
Discovery. It uses Apache ZooKeeper for master election, node fault
detection and state publishing. This plugin is still experimental and not
recommended to be used in production environments.

For more information visit plugin project page:
https://github.com/sonian/elasticsearch-zookeeper

--


(system) #9