JGroups and the Cloud


(Andrew Harvey-2) #1

Hi there,

I've been running ElasticSearch 0.6.0 for a little while now, and awaiting the new cloud plugin in 0.7.1.

I'm now trying to work out how to migrate my cluster over to the cloud module because I've come across a little trouble. Currently my nodes are running 0.6.0, using jgroups for discovery, with gateway, and I want to switch to cloud for both, but I'd really rather not shut down my cluster to do this, but to be able to go through a process something like this:

  • join a node which is configured to use the cloud for its gateway to the cluster,
  • remove one of the existing nodes, so the shards moved are moved to the new machine,
  • shut down that new machine and reconfigure it to use the cloud for discovery
  • start up a second machine configured to use the cloud for discovery and as a gateway
  • when I have verified the migration, shut down the old 0.6.0 machine
  • drink.

I'm struggling, however to execute the first step, because I can't seem to get the new machine to use jgroups to connect to the cluster.

I was also wondering if there might be a better way. This was all I could come up with on a Monday morning :slight_smile:

I'd appreciate any thoughts that people might have.

Andrew
Andrew Harvey / Developer
lexer
m/
t/ +61 2 9019 6379
w/ http://lexer.com.au
Help put an end to whaling. Visit http://www.givewhalesavoice.com.au/

Please consider the environment before printing this email
This email transmission is confidential and intended solely for the person or organisation to whom it is addressed. If you are not the intended recipient, you must not copy, distribute or disseminate the information, or take any action in relation to it and please delete this e-mail. Any views expressed in this message are those of the individual sender, except where the send specifically states them to be the views of any organisation or employer. If you have received this message in error, do not open any attachment but please notify the sender (above). This message has been checked for all known viruses powered by McAfee.

For further information visit http://www.mcafee.com/us/threat_center/default.asp
Please rely on your own virus check as no responsibility is taken by the sender for any damage rising out of any virus infection this communication may contain.

This message has been scanned for malware by Websense. www.websense.com


(Shay Banon) #2

Hi mate,

Sadly, you can't do that... . Moving to the cloud plugin requires a full
cluster restart. Sorry.

cheers,
shay.banon

On Mon, May 17, 2010 at 5:31 AM, Andrew Harvey
Andrew.Harvey@lexer.com.auwrote:

Hi there,

I've been running ElasticSearch 0.6.0 for a little while now, and awaiting
the new cloud plugin in 0.7.1.

I'm now trying to work out how to migrate my cluster over to the cloud
module because I've come across a little trouble. Currently my nodes are
running 0.6.0, using jgroups for discovery, with gateway, and I want to
switch to cloud for both, but I'd really rather not shut down my cluster to
do this, but to be able to go through a process something like this:

  • join a node which is configured to use the cloud for its gateway to the
    cluster,
  • remove one of the existing nodes, so the shards moved are moved to the
    new machine,
  • shut down that new machine and reconfigure it to use the cloud for
    discovery
  • start up a second machine configured to use the cloud for discovery and
    as a gateway
  • when I have verified the migration, shut down the old 0.6.0 machine
  • drink.

I'm struggling, however to execute the first step, because I can't seem to
get the new machine to use jgroups to connect to the cluster.

I was also wondering if there might be a better way. This was all I could
come up with on a Monday morning :slight_smile:

I'd appreciate any thoughts that people might have.

Andrew
Andrew Harvey / Developer
lexer
m/
t/ +61 2 9019 6379
w/ http://lexer.com.au
Help put an end to whaling. Visit http://www.givewhalesavoice.com.au/

Please consider the environment before printing this email
This email transmission is confidential and intended solely for the person
or organisation to whom it is addressed. If you are not the intended
recipient, you must not copy, distribute or disseminate the information, or
take any action in relation to it and please delete this e-mail. Any views
expressed in this message are those of the individual sender, except where
the send specifically states them to be the views of any organisation or
employer. If you have received this message in error, do not open any
attachment but please notify the sender (above). This message has been
checked for all known viruses powered by McAfee.

For further information visit
http://www.mcafee.com/us/threat_center/default.asp
Please rely on your own virus check as no responsibility is taken by the
sender for any damage rising out of any virus infection this communication
may contain.

This message has been scanned for malware by Websense. www.websense.com


(Berkay Mollamustafaoglu-2) #3

While the subject is opened, what would be a good approach/process to move
data from one version to another that would minimize downtime?

Some way of using the write behind gw files? replication between different
ES clusters?

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Mon, May 17, 2010 at 4:07 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi mate,

Sadly, you can't do that... . Moving to the cloud plugin requires a full
cluster restart. Sorry.

cheers,
shay.banon

On Mon, May 17, 2010 at 5:31 AM, Andrew Harvey <Andrew.Harvey@lexer.com.au

wrote:

Hi there,

I've been running ElasticSearch 0.6.0 for a little while now, and awaiting
the new cloud plugin in 0.7.1.

I'm now trying to work out how to migrate my cluster over to the cloud
module because I've come across a little trouble. Currently my nodes are
running 0.6.0, using jgroups for discovery, with gateway, and I want to
switch to cloud for both, but I'd really rather not shut down my cluster to
do this, but to be able to go through a process something like this:

  • join a node which is configured to use the cloud for its gateway to the
    cluster,
  • remove one of the existing nodes, so the shards moved are moved to the
    new machine,
  • shut down that new machine and reconfigure it to use the cloud for
    discovery
  • start up a second machine configured to use the cloud for discovery and
    as a gateway
  • when I have verified the migration, shut down the old 0.6.0 machine
  • drink.

I'm struggling, however to execute the first step, because I can't seem to
get the new machine to use jgroups to connect to the cluster.

I was also wondering if there might be a better way. This was all I could
come up with on a Monday morning :slight_smile:

I'd appreciate any thoughts that people might have.

Andrew
Andrew Harvey / Developer
lexer
m/
t/ +61 2 9019 6379
w/ http://lexer.com.au
Help put an end to whaling. Visit http://www.givewhalesavoice.com.au/

Please consider the environment before printing this email
This email transmission is confidential and intended solely for the person
or organisation to whom it is addressed. If you are not the intended
recipient, you must not copy, distribute or disseminate the information, or
take any action in relation to it and please delete this e-mail. Any views
expressed in this message are those of the individual sender, except where
the send specifically states them to be the views of any organisation or
employer. If you have received this message in error, do not open any
attachment but please notify the sender (above). This message has been
checked for all known viruses powered by McAfee.

For further information visit
http://www.mcafee.com/us/threat_center/default.asp
Please rely on your own virus check as no responsibility is taken by the
sender for any damage rising out of any virus infection this communication
may contain.

This message has been scanned for malware by Websense. www.websense.com


(Shay Banon) #4

In general, the gateway format should not change, so, you can shut down the
cluster and then start it. That does entail downtime though. Another option
is to start another cluster and migrate the data.

In later versions, I plan to work on building support for different versions
to coexist. Not simple as not just wire format will change, but also
behavior...

cheers,
shay.banon

On Mon, May 17, 2010 at 6:01 PM, Berkay Mollamustafaoglu
mberkay@gmail.comwrote:

While the subject is opened, what would be a good approach/process to move
data from one version to another that would minimize downtime?

Some way of using the write behind gw files? replication between different
ES clusters?

Regards,
Berkay Mollamustafaoglu
mberkay on yahoo, google and skype

On Mon, May 17, 2010 at 4:07 AM, Shay Banon shay.banon@elasticsearch.comwrote:

Hi mate,

Sadly, you can't do that... . Moving to the cloud plugin requires a full
cluster restart. Sorry.

cheers,
shay.banon

On Mon, May 17, 2010 at 5:31 AM, Andrew Harvey <
Andrew.Harvey@lexer.com.au> wrote:

Hi there,

I've been running ElasticSearch 0.6.0 for a little while now, and
awaiting the new cloud plugin in 0.7.1.

I'm now trying to work out how to migrate my cluster over to the cloud
module because I've come across a little trouble. Currently my nodes are
running 0.6.0, using jgroups for discovery, with gateway, and I want to
switch to cloud for both, but I'd really rather not shut down my cluster to
do this, but to be able to go through a process something like this:

  • join a node which is configured to use the cloud for its gateway to
    the cluster,
  • remove one of the existing nodes, so the shards moved are moved to the
    new machine,
  • shut down that new machine and reconfigure it to use the cloud for
    discovery
  • start up a second machine configured to use the cloud for discovery
    and as a gateway
  • when I have verified the migration, shut down the old 0.6.0 machine
  • drink.

I'm struggling, however to execute the first step, because I can't seem
to get the new machine to use jgroups to connect to the cluster.

I was also wondering if there might be a better way. This was all I could
come up with on a Monday morning :slight_smile:

I'd appreciate any thoughts that people might have.

Andrew
Andrew Harvey / Developer
lexer
m/
t/ +61 2 9019 6379
w/ http://lexer.com.au
Help put an end to whaling. Visit http://www.givewhalesavoice.com.au/


Please consider the environment before printing this email
This email transmission is confidential and intended solely for the
person or organisation to whom it is addressed. If you are not the intended
recipient, you must not copy, distribute or disseminate the information, or
take any action in relation to it and please delete this e-mail. Any views
expressed in this message are those of the individual sender, except where
the send specifically states them to be the views of any organisation or
employer. If you have received this message in error, do not open any
attachment but please notify the sender (above). This message has been
checked for all known viruses powered by McAfee.

For further information visit
http://www.mcafee.com/us/threat_center/default.asp
Please rely on your own virus check as no responsibility is taken by the
sender for any damage rising out of any virus infection this communication
may contain.

This message has been scanned for malware by Websense. www.websense.com


(system) #5