ANN: new elasticsearch discovery plugin - eskka


(shikhar) #1

See README https://github.com/shikhar/eskka/blob/release-0.1/README.md

I'd love to have feedback on this first release!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHWG4DMdPDCSpm0QJOznZuMDgi%2BTdFGx8dBTXvsACLuTvFmgxg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(shikhar) #2

Just released 0.1.1 https://github.com/shikhar/eskka/releases/tag/v0.1.1

This version is working well in my manual testing. Automated testing is on
the roadmap...

On Mon, May 5, 2014 at 10:49 AM, shikhar shikhar@schmizz.net wrote:

See README https://github.com/shikhar/eskka/blob/release-0.1/README.md

I'd love to have feedback on this first release!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHWG4DNiwCxbZakzogFfqFxYxabcQ_ysG2_OMd06%3D%2BfDqFEQdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Ivan Brusic) #3

Extremely interesting! What is the recommended size of the Akka cluster
compared to the Elasticsearch cluster?

--
Ivan

On Tue, May 6, 2014 at 8:42 PM, shikhar shikhar@schmizz.net wrote:

Just released 0.1.1 https://github.com/shikhar/eskka/releases/tag/v0.1.1

This version is working well in my manual testing. Automated testing is on
the roadmap...

On Mon, May 5, 2014 at 10:49 AM, shikhar shikhar@schmizz.net wrote:

See README https://github.com/shikhar/eskka/blob/release-0.1/README.md

I'd love to have feedback on this first release!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHWG4DNiwCxbZakzogFfqFxYxabcQ_ysG2_OMd06%3D%2BfDqFEQdA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAHWG4DNiwCxbZakzogFfqFxYxabcQ_ysG2_OMd06%3D%2BfDqFEQdA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDjg7Uaw4tTVr2-8cGM7w2sutH6F2XzCVj3JjKK-RCt5g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(shikhar) #4

All Elasticsearch nodes will end up being part of the Akka cluster :slight_smile: I
think you're really asking how many seed nodes you should specify. The seed
node list is probably going to be similar to what you might use for
zen.unicast.hosts.

Worth noting that besides being initial contact points for when the cluster
is starting up, with eskka they are also used for resolving partitions.
Given this requirement, you would ideally have 3 or more specified. It is
perfectly ok to have all of your nodes listed, if you know their addresses
before startup.

On Wed, May 7, 2014 at 9:31 PM, Ivan Brusic ivan@brusic.com wrote:

Extremely interesting! What is the recommended size of the Akka cluster
compared to the Elasticsearch cluster?

--
Ivan

On Tue, May 6, 2014 at 8:42 PM, shikhar shikhar@schmizz.net wrote:

Just released 0.1.1https://github.com/shikhar/eskka/releases/tag/v0.1.1

This version is working well in my manual testing. Automated testing is
on the roadmap...

On Mon, May 5, 2014 at 10:49 AM, shikhar shikhar@schmizz.net wrote:

See README https://github.com/shikhar/eskka/blob/release-0.1/README.md

I'd love to have feedback on this first release!

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHWG4DNiwCxbZakzogFfqFxYxabcQ_ysG2_OMd06%3D%2BfDqFEQdA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAHWG4DNiwCxbZakzogFfqFxYxabcQ_ysG2_OMd06%3D%2BfDqFEQdA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDjg7Uaw4tTVr2-8cGM7w2sutH6F2XzCVj3JjKK-RCt5g%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDjg7Uaw4tTVr2-8cGM7w2sutH6F2XzCVj3JjKK-RCt5g%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHWG4DO-nGqKuv9i_wFpR0nomTY4mk%3DmkVxvgai3bjO8sSyRxw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(shikhar) #5

On Thu, May 8, 2014 at 1:02 PM, shikhar shikhar@schmizz.net wrote:

Worth noting that besides being initial contact points for when the
cluster is starting up, with eskka they are also used for resolving
partitions. Given this requirement, you would ideally have 3 or more
specified. It is perfectly ok to have all of your nodes listed, if you know
their addresses before startup.

Another idea for what nodes to use as seed: if you are using master-only
nodes, make them seed nodes.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHWG4DPFje-U8LriQgDMefhUzxOr8CuH1jqKghqMDO3bLYgpvg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Otis Gospodnetić) #6

Cool, Shikhar,

At Sematext we use both ES and Akka (in SPM http://sematext.com/spm/), so
this is interesting for me to see... Would it make sense to add a bit more
to the README..... things like:

  • why? is something wrong with Zen?
  • pros and cons of this vs. Zen vs. ZK

Thanks,
Otis

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Thursday, May 8, 2014 3:32:24 AM UTC-4, Shikhar Bhushan wrote:

All Elasticsearch nodes will end up being part of the Akka cluster :slight_smile: I
think you're really asking how many seed nodes you should specify. The seed
node list is probably going to be similar to what you might use for
zen.unicast.hosts.

Worth noting that besides being initial contact points for when the
cluster is starting up, with eskka they are also used for resolving
partitions. Given this requirement, you would ideally have 3 or more
specified. It is perfectly ok to have all of your nodes listed, if you know
their addresses before startup.

https://github.com/shikhar/eskka#configuration

On Wed, May 7, 2014 at 9:31 PM, Ivan Brusic <iv...@brusic.com<javascript:>

wrote:

Extremely interesting! What is the recommended size of the Akka cluster
compared to the Elasticsearch cluster?

--
Ivan

On Tue, May 6, 2014 at 8:42 PM, shikhar <shi...@schmizz.net <javascript:>

wrote:

Just released 0.1.1https://github.com/shikhar/eskka/releases/tag/v0.1.1

This version is working well in my manual testing. Automated testing is
on the roadmap...

On Mon, May 5, 2014 at 10:49 AM, shikhar <shi...@schmizz.net<javascript:>

wrote:

See READMEhttps://github.com/shikhar/eskka/blob/release-0.1/README.md

I'd love to have feedback on this first release!

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHWG4DNiwCxbZakzogFfqFxYxabcQ_ysG2_OMd06%3D%2BfDqFEQdA%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAHWG4DNiwCxbZakzogFfqFxYxabcQ_ysG2_OMd06%3D%2BfDqFEQdA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDjg7Uaw4tTVr2-8cGM7w2sutH6F2XzCVj3JjKK-RCt5g%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDjg7Uaw4tTVr2-8cGM7w2sutH6F2XzCVj3JjKK-RCt5g%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9b73c46d-66dd-4c29-9e76-feba3f033133%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(shikhar) #7

Thanks for the feedback Otis!

On Fri, May 9, 2014 at 12:33 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

At Sematext we use both ES and Akka (in SPM http://sematext.com/spm/),
so this is interesting for me to see... Would it make sense to add a bit
more to the README..... things like:

  • why? is something wrong with Zen?
  • pros and cons of this vs. Zen vs. ZK

Makes sense to add this context, I'll add to the docs soon.

The major con for eskka is of course that it is new and unproven.

vs zookeeper

You don't need to setup or admin ZooKeeper.

In Shay's resiliency blog post <http://resiliency and elasticsearch> he
writes "By having the discovery module in Elasticsearch using its own
infrastructure, specifically the communication layer, it means that we can
use the “liveness” of the cluster to assess its health." This is an
interesting point which partly holds for eskka as well, since it also lives
in the same JVM. However it doesn't use the same communication layer (the
ES internal transport), the akka cluster runs on a different port.

vs zen

Essentially: eskka is built to be split-brain resistant. But then so is Zen
in theory :wink: So I'd like to maybe do some testing with Jepsen before
making that tall claim.

Both Zen and Akka Cluster are implementing a gossip protocol for discovery
& fault-detection, however to me the latter feels better specified and
tested. eskka itself does not have any tests at the application-level which
I intend to fix.

I'd like to think the use of actors -- the main ones being
Masterhttps://github.com/shikhar/eskka/blob/master/src/main/scala/eskka/Master.scala
, Followerhttps://github.com/shikhar/eskka/blob/master/src/main/scala/eskka/Follower.scala
and QuorumBasedPartitionMonitorhttps://github.com/shikhar/eskka/blob/master/src/main/scala/eskka/QuorumBasedPartitionMonitor.scala

makes it easier to reason about state in a distributed system and maintain
invariants.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHWG4DMMPZMhmi7w74vsUqJJria%2BtVj5p7sn-v%3DVeXNUqaXziw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #8