Quit the quorum


(Simon Even) #1

Good day!

I've developed an elasticsearch plugin which wathces the index changes and
broadcasts them to a websocket running within the plugin. And ofcourse I
ran into a problem :slight_smile: If i use a cluster of 3 (for example) elasticsearch
services, writing a new document won't trigger the index change event on
all es servers but only on those having the shard. My questing is how can I
force es to write all the shards to all the servers at all times but at the
same time keep the scaleability? By keeping the scaleability I mean an
option to add more servers without the need of changing the config files
with number of replicas and restarting the services.

I've tried playing with write_consistency but I believe that's totally
different thing.

Thanks in advance!
-Simon

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/3a05aaf0-f13f-4447-a598-9812f2672972%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Jörg Prante) #2

In my websocket plugin, I solve this by a pub/sub architecture

  • each node with websocket plugin opens a channel to every other node with
    websocket plugin
  • a publisher offers a service to which other clients can subscribe
  • the result is a service where a number of subscribing clients are
    permanently connected and wait for messages
  • if a client pushes a document, he can also publish a success message to
    the service
  • every subscribing client is notified about the message (implemented as
    scan/scroll action over a dedicated pub/sub index)

Jörg

On Fri, Apr 4, 2014 at 9:39 AM, Simon Even even.simon@gmail.com wrote:

Good day!

I've developed an elasticsearch plugin which wathces the index changes and
broadcasts them to a websocket running within the plugin. And ofcourse I
ran into a problem :slight_smile: If i use a cluster of 3 (for example) elasticsearch
services, writing a new document won't trigger the index change event on
all es servers but only on those having the shard. My questing is how can I
force es to write all the shards to all the servers at all times but at the
same time keep the scaleability? By keeping the scaleability I mean an
option to add more servers without the need of changing the config files
with number of replicas and restarting the services.

I've tried playing with write_consistency but I believe that's totally
different thing.

Thanks in advance!
-Simon

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3a05aaf0-f13f-4447-a598-9812f2672972%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3a05aaf0-f13f-4447-a598-9812f2672972%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsnJGMk%2BhTLVDqQ2hTvqEr9vjnooX8pa-grvcp9C9eag%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Simon Even) #3

Jörg, thanks for your post!

My idea is to have a cluster of es with a plugin and an nginx above it to
balance the sockets. In this case each plugin has to broadcast all changes
made in the es,
for this purpose I need to have all the shards on all the servers, or am I
wrong?

On Friday, April 4, 2014 3:28:21 PM UTC+4, Jörg Prante wrote:

In my websocket plugin, I solve this by a pub/sub architecture

  • each node with websocket plugin opens a channel to every other node with
    websocket plugin
  • a publisher offers a service to which other clients can subscribe
  • the result is a service where a number of subscribing clients are
    permanently connected and wait for messages
  • if a client pushes a document, he can also publish a success message to
    the service
  • every subscribing client is notified about the message (implemented as
    scan/scroll action over a dedicated pub/sub index)

https://github.com/jprante/elasticsearch-transport-websocket/

Jörg

On Fri, Apr 4, 2014 at 9:39 AM, Simon Even <even....@gmail.com<javascript:>

wrote:

Good day!

I've developed an elasticsearch plugin which wathces the index changes
and broadcasts them to a websocket running within the plugin. And ofcourse
I ran into a problem :slight_smile: If i use a cluster of 3 (for example) elasticsearch
services, writing a new document won't trigger the index change event on
all es servers but only on those having the shard. My questing is how can I
force es to write all the shards to all the servers at all times but at the
same time keep the scaleability? By keeping the scaleability I mean an
option to add more servers without the need of changing the config files
with number of replicas and restarting the services.

I've tried playing with write_consistency but I believe that's totally
different thing.

Thanks in advance!
-Simon

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/3a05aaf0-f13f-4447-a598-9812f2672972%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/3a05aaf0-f13f-4447-a598-9812f2672972%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f83e8eb4-d841-4201-acac-d174a4c66675%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4