1.4.0 data node can't join existing 1.3.4 cluster

Elastic Stack Elasticsearch
1

(using elasticsearch-cloud-aws 2.4)

This should work, right? Or do I need to upgrade the cluster to 1.3.5 first?

The connection fails after a few errors like:

2014-11-13 07:18:22,498 [WARN] org.elasticsearch.discovery.zen.ping.unicast

  • [Porcupine] failed to send ping to
    [[#cloud-i-b743e456-0][530-1d][inet[/10.186.145.210:9300]]]
    org.elasticsearch.transport.RemoteTransportException:
    [Nomad][inet[/10.186.145.210:9300]][internal:discovery/zen/unicast]
    Caused by: org.elasticsearch.transport.ActionNotFoundTransportException: No
    handler for action [internal:discovery/zen/unicast]
    at
    org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
    ~[org.elasticsearch.elasticsearch-1.4.0.jar:na]
    at
    org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)
    ~[org.elasticsearch.elasticsearch-1.4.0.jar:na]
    at
    org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
    ~[org.elasticsearch.elasticsearch-1.4.0.jar:na]

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/efd3232c-b22f-4f2b-94c4-c942b75e81bf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I'm seeing something very similar, also from 1.3.4 to 1.4.0, also using the
elasticsearch-cloud-aws plugin 2.4.

[2014-11-13 16:02:26,055][WARN ][discovery.zen.ping.unicast] [White Fang]
failed to send ping to
[[#cloud-i-03f79bcb-0][localhost][inet[/10.0.0.76:9300]]]
org.elasticsearch.transport.RemoteTransportException: [Kraven the
Hunter][inet[/10.0.0.76:9300]][internal:discovery/zen/unicast]
Caused by: org.elasticsearch.transport.ActionNotFoundTransportException: No
handler for action [internal:discovery/zen/unicast]
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Do not mix 1.3 with 1.4 nodes, it does not work.

Jörg

On Thu, Nov 13, 2014 at 5:16 PM, Todd Kamin todd@crowdtangle.com wrote:

I'm seeing something very similar, also from 1.3.4 to 1.4.0, also using
the elasticsearch-cloud-aws plugin 2.4.

[2014-11-13 16:02:26,055][WARN ][discovery.zen.ping.unicast] [White Fang]
failed to send ping to
[[#cloud-i-03f79bcb-0][localhost][inet[/10.0.0.76:9300]]]
org.elasticsearch.transport.RemoteTransportException: [Kraven the
Hunter][inet[/10.0.0.76:9300]][internal:discovery/zen/unicast]
Caused by: org.elasticsearch.transport.ActionNotFoundTransportException:
No handler for action [internal:discovery/zen/unicast]
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGOOeMcuZqw7xADCuTc-MTQkuqQ0QtnBR%2Bg9TWBS6DOng%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

4

Rolling upgrades should be supported:

How else can you perform a rolling upgrade without having a mixed cluster?

--
Ivan

On Thu, Nov 13, 2014 at 1:05 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Do not mix 1.3 with 1.4 nodes, it does not work.

Jörg

On Thu, Nov 13, 2014 at 5:16 PM, Todd Kamin todd@crowdtangle.com wrote:

I'm seeing something very similar, also from 1.3.4 to 1.4.0, also using
the elasticsearch-cloud-aws plugin 2.4.

[2014-11-13 16:02:26,055][WARN ][discovery.zen.ping.unicast] [White Fang]
failed to send ping to
[[#cloud-i-03f79bcb-0][localhost][inet[/10.0.0.76:9300]]]
org.elasticsearch.transport.RemoteTransportException: [Kraven the
Hunter][inet[/10.0.0.76:9300]][internal:discovery/zen/unicast]
Caused by: org.elasticsearch.transport.ActionNotFoundTransportException:
No handler for action [internal:discovery/zen/unicast]
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGOOeMcuZqw7xADCuTc-MTQkuqQ0QtnBR%2Bg9TWBS6DOng%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGOOeMcuZqw7xADCuTc-MTQkuqQ0QtnBR%2Bg9TWBS6DOng%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDrfhTAc6FhPpU-ZcHtW4Qp_9qB9G6z%2BSXUR7fqtB52_A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

I discovered a peculiarity, well documented in the source

/**
 * when pinging the initial configured target hosts, we do not know

their version. We therefore use
* the lowest possible version (i.e., 1.0.0) for serializing
information on the wire. As of 1.4, we needed to extend
* the information sent in a ping, to prefer nodes which have
previously joined the cluster during master election.
* This information is only needed if all the cluster is on version 1.4
or up. To bypass this issue we introduce
* a second action name which is guaranteed to exist only on nodes from
version 1.4.0 and up. Using this action,
* we can safely use 1.4.0 as a serialization format. If this fails
with a {@link ActionNotFoundTransportException}
* we know we speak to a node with <1.4 version, and fall back to use
{@link #ACTION_NAME}.
*/
public static final String ACTION_NAME_GTE_1_4 =
"internal:discovery/zen/unicast_gte_1_4";

Not sure if everything works well...

Jörg

On Thu, Nov 13, 2014 at 10:17 PM, Ivan Brusic ivan@brusic.com wrote:

Rolling upgrades should be supported:

Elasticsearch Platform — Find real-time answers at scale | Elastic

How else can you perform a rolling upgrade without having a mixed cluster?

--
Ivan

On Thu, Nov 13, 2014 at 1:05 PM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Do not mix 1.3 with 1.4 nodes, it does not work.

Jörg

On Thu, Nov 13, 2014 at 5:16 PM, Todd Kamin todd@crowdtangle.com wrote:

I'm seeing something very similar, also from 1.3.4 to 1.4.0, also using
the elasticsearch-cloud-aws plugin 2.4.

[2014-11-13 16:02:26,055][WARN ][discovery.zen.ping.unicast] [White
Fang] failed to send ping to
[[#cloud-i-03f79bcb-0][localhost][inet[/10.0.0.76:9300]]]
org.elasticsearch.transport.RemoteTransportException: [Kraven the
Hunter][inet[/10.0.0.76:9300]][internal:discovery/zen/unicast]
Caused by: org.elasticsearch.transport.ActionNotFoundTransportException:
No handler for action [internal:discovery/zen/unicast]
at
org.elasticsearch.transport.netty.MessageChannelHandler.handleRequest(MessageChannelHandler.java:210)
at
org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:111)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/840f26f7-c401-4229-b94d-092ee2ef4974%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGOOeMcuZqw7xADCuTc-MTQkuqQ0QtnBR%2Bg9TWBS6DOng%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGOOeMcuZqw7xADCuTc-MTQkuqQ0QtnBR%2Bg9TWBS6DOng%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDrfhTAc6FhPpU-ZcHtW4Qp_9qB9G6z%2BSXUR7fqtB52_A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQDrfhTAc6FhPpU-ZcHtW4Qp_9qB9G6z%2BSXUR7fqtB52_A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFN06R7-sYGcsaUtg_VagcjSHikbmPw%3DUSz-H29nvrmWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

On Thu, Nov 13, 2014 at 10:05 AM, joergprante@gmail.com
joergprante@gmail.com wrote:

Do not mix 1.3 with 1.4 nodes, it does not work.

If that is so, that seems like something the release notes should mention?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

I think this is only related to unicast. But, nevertheless, it should
work... not sure if this is a bug or a feature....

Jörg

On Fri, Nov 14, 2014 at 12:58 AM, Eric Jain eric.jain@gmail.com wrote:

On Thu, Nov 13, 2014 at 10:05 AM, joergprante@gmail.com
joergprante@gmail.com wrote:

Do not mix 1.3 with 1.4 nodes, it does not work.

If that is so, that seems like something the release notes should mention?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoE7GvQiiz2G-PjNWXNKg5NWHE%2BUfwE0bg_mHekw6e5WZw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

8

I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster
with cloud-aws plugin version 2.4.0. Is there a workaround to use during
upgrade, since I assume it's not a problem when they're all upgraded to
1.4.0.

On Friday, November 14, 2014 11:33:45 AM UTC+1, Jörg Prante wrote:

I think this is only related to unicast. But, nevertheless, it should
work... not sure if this is a bug or a feature....

Jörg

On Fri, Nov 14, 2014 at 12:58 AM, Eric Jain <eric...@gmail.com
<javascript:>> wrote:

On Thu, Nov 13, 2014 at 10:05 AM, joerg...@gmail.com <javascript:>
<joerg...@gmail.com <javascript:>> wrote:

Do not mix 1.3 with 1.4 nodes, it does not work.

If that is so, that seems like something the release notes should mention?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/23b1428c-1e55-498e-ac8f-499adc8528e0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

9

1.4.0 trying to join a 1.3.5 cluster with cloud-aws also fails.

On Friday, November 14, 2014 12:41:08 PM UTC+1, madsm...@colourbox.com
wrote:

I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4
cluster with cloud-aws plugin version 2.4.0. Is there a workaround to use
during upgrade, since I assume it's not a problem when they're all upgraded
to 1.4.0.

On Friday, November 14, 2014 11:33:45 AM UTC+1, Jörg Prante wrote:

I think this is only related to unicast. But, nevertheless, it should
work... not sure if this is a bug or a feature....

Jörg

On Fri, Nov 14, 2014 at 12:58 AM, Eric Jain eric...@gmail.com wrote:

On Thu, Nov 13, 2014 at 10:05 AM, joerg...@gmail.com
joerg...@gmail.com wrote:

Do not mix 1.3 with 1.4 nodes, it does not work.

If that is so, that seems like something the release notes should
mention?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BKyW3nZ82uf1w1C_i3O9oW%3D%3DhG7f3WUC2-g40%3DwUn5Fgw%40mail.gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/967ce7d9-3f53-4478-868b-85443a904620%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

10

On Fri, Nov 14, 2014 at 3:41 AM, madsmartin@colourbox.com wrote:

I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4 cluster
with cloud-aws plugin version 2.4.0. Is there a workaround to use during
upgrade, since I assume it's not a problem when they're all upgraded to
1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHte5%2BJ%2BfnBnus%3DOX6itdwcaB9%2Bh_KDMwDYRBWU-4fWL0CohJA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

11

As soon as we surf the internet and looking for the jammer products we can see that there are various types of signal jammers that are for sale in the market now such as the mobile phone jammers, GPS jammers, wifi jammers, UHF jammers, video camera jamming device, the multi-functional signal jammers and so many others kinds of signal jammers for sale as well. And what people now need to do is just select the best one according to their needs.

12

We are running a 1.3.4 cluster using the AWS plugin and I noticed the same
error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what
would happen if I upgraded a 2nd node. Would it split into 2 clusters, have
the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined
the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you try
with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:

On Fri, Nov 14, 2014 at 3:41 AM, <madsm...@colourbox.com <javascript:>>
wrote:

I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4
cluster
with cloud-aws plugin version 2.4.0. Is there a workaround to use during
upgrade, since I assume it's not a problem when they're all upgraded to
1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f618a79d-0a4f-4f97-8b89-ab6ccb9d1cbe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

13

I stand corrected, this did not work on our main cluster.

On Monday, 17 November 2014 11:13:22 UTC, Matthew Barrington wrote:

We are running a 1.3.4 cluster using the AWS plugin and I noticed the same
error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what
would happen if I upgraded a 2nd node. Would it split into 2 clusters, have
the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined
the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you try
with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:

On Fri, Nov 14, 2014 at 3:41 AM, madsm...@colourbox.com wrote:

I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4
cluster
with cloud-aws plugin version 2.4.0. Is there a workaround to use
during
upgrade, since I assume it's not a problem when they're all upgraded to
1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/fc79c5e7-ddd6-4f52-9641-1bd01df3b866%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

14

I found this thread while trying to research the same issue and it looks like there is currently no resolution. We like to keep up on our elasticsearch upgrades as often as possible and do rolling upgrades to keep our clusters up. When testing I'm having the same issue, I cannot add a 1.4.0 box to the existing 1.3.4 cluster.

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CF8216AA982AF47A8E6DEACA629D22B4EBEEF75%40s-us-ex-6.US.R5S.com.
For more options, visit https://groups.google.com/d/optout.

15

Hi Christian,

I'm not sure what thread you refer to exactly, but this shouldn't happen.
Can you describe the problem you have some more? Anything in the nodes?
(both the 1.4 node and the master)

Cheers,
Boaz

On Wednesday, November 19, 2014 2:39:57 AM UTC+1, Christian Hedegaard wrote:

I found this thread while trying to research the same issue and it looks
like there is currently no resolution. We like to keep up on our
elasticsearch upgrades as often as possible and do rolling upgrades to keep
our clusters up. When testing I’m having the same issue, I cannot add a
1.4.0 box to the existing 1.3.4 cluster.

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/16f33a69-fb07-4db1-9c05-b9031c867a63%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

16

From the archives: https://groups.google.com/forum/#!searchin/elasticsearch/1.4$20data$20node/elasticsearch/8pUwFld88tI/sBH7bB7rYzsJ

Same subject as mine. Started on the 13th.

Anyways, I’m having the exact same issue. I’ve got a cluster on 1.3.4 (well, now I’ve upgraded it to 1.3.5). When I provision a new node with 1.4 and the cloud-aws plugin with the proper version (2.4.0), the new node will not come up and join the cluster and so a rolling upgrade does not appear possible.

From: Boaz Leskes [mailto:b.leskes@gmail.com]
Sent: Wednesday, November 19, 2014 2:27 PM
To: elasticsearch@googlegroups.com
Cc: Christian Hedegaard
Subject: Re: 1.4.0 data node can't join existing 1.3.4 cluster

Hi Christian,

I'm not sure what thread you refer to exactly, but this shouldn't happen. Can you describe the problem you have some more? Anything in the nodes? (both the 1.4 node and the master)

Cheers,
Boaz

On Wednesday, November 19, 2014 2:39:57 AM UTC+1, Christian Hedegaard wrote:
I found this thread while trying to research the same issue and it looks like there is currently no resolution. We like to keep up on our elasticsearch upgrades as often as possible and do rolling upgrades to keep our clusters up. When testing I’m having the same issue, I cannot add a 1.4.0 box to the existing 1.3.4 cluster.

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CF8216AA982AF47A8E6DEACA629D22B4EBF0DA6%40s-us-ex-6.US.R5S.com.
For more options, visit https://groups.google.com/d/optout.

17

We use GCE and the GCE plugin, on initial load it seemed like what you are
describing. Only when I examined the logs carefully i noticed that its
more strict in reading the yaml and in our case the zone used to be

cloud:
gce:
zone: ["us-central1-a","us-central1-f"]

And with 1.4 it should now be:

cloud:
gce:
zone: us-central1-a,us-central1-f

I haven't looked into the AWS plugin, but perhaps it the same kind of
strict yaml configuration read.

Good luck!

On Wednesday, November 19, 2014 5:43:49 PM UTC-5, Christian Hedegaard wrote:

From the archives:
Redirecting to Google Groups

Same subject as mine. Started on the 13th.

Anyways, I’m having the exact same issue. I’ve got a cluster on 1.3.4
(well, now I’ve upgraded it to 1.3.5). When I provision a new node with 1.4
and the cloud-aws plugin with the proper version (2.4.0), the new node will
not come up and join the cluster and so a rolling upgrade does not appear
possible.

From: Boaz Leskes [mailto:b.le...@gmail.com <javascript:>]
Sent: Wednesday, November 19, 2014 2:27 PM
To: elasti...@googlegroups.com <javascript:>
Cc: Christian Hedegaard
Subject: Re: 1.4.0 data node can't join existing 1.3.4 cluster

Hi Christian,

I'm not sure what thread you refer to exactly, but this shouldn't happen.
Can you describe the problem you have some more? Anything in the nodes?
(both the 1.4 node and the master)

Cheers,

Boaz

On Wednesday, November 19, 2014 2:39:57 AM UTC+1, Christian Hedegaard
wrote:

I found this thread while trying to research the same issue and it looks
like there is currently no resolution. We like to keep up on our
elasticsearch upgrades as often as possible and do rolling upgrades to keep
our clusters up. When testing I’m having the same issue, I cannot add a
1.4.0 box to the existing 1.3.4 cluster.

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/294e7c85-10a0-4975-a3ef-12fd106cd69c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

18

FYI, I have found a solution that works (at least for me).

I’ve got a small cluster for testing, only 4 v1.3.5 nodes. What I’ve done is bring up 4X new v1.4.0 nodes as data-only machines. In the yaml I added a line to point the nodes via unicast explicitly to the current master:
discovery.zen.ping.unicast.hosts: ["10.210.9.224:9300"]

When I restarted elasticsearch with that setting, with cloud-aws installed and configured on version 2.4.0, the new nodes found the cluster and properly joined it.

I will now start nuking the old v1.3.5 nodes to migrate the data off of them. Before the final 1.3.5 node is nuked, I will change the config on one of the v1.4.0 nodes to allow it as master and restart it.

I’m not sure if the master stuff is needed or not, but I was very afraid of a split-brain problem. I have another 4-node testing cluster that I will be able to try this upgrade again with in a more controlled manner.

I’m NOT looking forward to upgrading our current production cluster this way (15 data-only nodes, 3 master-only nodes).

So it would appear that the problem is somewhere in the unicast discovery code. The question is who’s to blame? Elasticsearch or the cloud-aws plugin?

From: Boaz Leskes [mailto:b.leskes@gmail.com]
Sent: Wednesday, November 19, 2014 2:27 PM
To: elasticsearch@googlegroups.com
Cc: Christian Hedegaard
Subject: Re: 1.4.0 data node can't join existing 1.3.4 cluster

Hi Christian,

I'm not sure what thread you refer to exactly, but this shouldn't happen. Can you describe the problem you have some more? Anything in the nodes? (both the 1.4 node and the master)

Cheers,
Boaz

On Wednesday, November 19, 2014 2:39:57 AM UTC+1, Christian Hedegaard wrote:
I found this thread while trying to research the same issue and it looks like there is currently no resolution. We like to keep up on our elasticsearch upgrades as often as possible and do rolling upgrades to keep our clusters up. When testing I’m having the same issue, I cannot add a 1.4.0 box to the existing 1.3.4 cluster.

Is there a fix for this anticipated?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5CF8216AA982AF47A8E6DEACA629D22B4EBF409B%40s-us-ex-6.US.R5S.com.
For more options, visit https://groups.google.com/d/optout.

19

Is there any of the elasticsearch team members that can hint to whether or
not this is something that will be fixed in 1.4.1? Then we'll simply wait
for it instead of doing different hacks to upgrade.

On Monday, November 17, 2014 12:35:03 PM UTC+1, Matthew Barrington wrote:

I stand corrected, this did not work on our main cluster.

On Monday, 17 November 2014 11:13:22 UTC, Matthew Barrington wrote:

We are running a 1.3.4 cluster using the AWS plugin and I noticed the
same error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what
would happen if I upgraded a 2nd node. Would it split into 2 clusters, have
the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined
the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you try
with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:

On Fri, Nov 14, 2014 at 3:41 AM, madsm...@colourbox.com wrote:

I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4
cluster
with cloud-aws plugin version 2.4.0. Is there a workaround to use
during
upgrade, since I assume it's not a problem when they're all upgraded
to
1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at zenobase.com.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/312dcdc1-d826-4cb9-b480-620232634ea7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

20

It's being looked at, but I don't know much beyond that at the moment sorry.

On 21 November 2014 20:02, madsmartin@colourbox.com wrote:

Is there any of the elasticsearch team members that can hint to whether or
not this is something that will be fixed in 1.4.1? Then we'll simply wait
for it instead of doing different hacks to upgrade.

On Monday, November 17, 2014 12:35:03 PM UTC+1, Matthew Barrington wrote:

I stand corrected, this did not work on our main cluster.

On Monday, 17 November 2014 11:13:22 UTC, Matthew Barrington wrote:

We are running a 1.3.4 cluster using the AWS plugin and I noticed the
same error when I tried to upgrade a single node.

Since I was trying this on my test cluster first I decided to see what
would happen if I upgraded a 2nd node. Would it split into 2 clusters, have
the same issue, etc.

What I discovered was that when 2 nodes were upgraded to 1.4 they joined
the cluster correctly and everything looks to be working.

SO the problem seems to be for the initial node to join, but when you
try with two everything works out.

On Friday, 14 November 2014 18:05:01 UTC, Eric Jain wrote:

On Fri, Nov 14, 2014 at 3:41 AM, madsm...@colourbox.com wrote:

I'm also seing this problem when a 1.4.0 node tries joining a 1.3.4
cluster
with cloud-aws plugin version 2.4.0. Is there a workaround to use
during
upgrade, since I assume it's not a problem when they're all upgraded
to
1.4.0.

I ended up starting a new cluster (ignoring all the warnings logged on
startup), and restoring from a snapshot. Once all the 1.3.4 nodes were
gone, no issues.

--
Eric Jain
Got data? Get answers at zenobase.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/312dcdc1-d826-4cb9-b480-620232634ea7%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/312dcdc1-d826-4cb9-b480-620232634ea7%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAF3ZnZn-ryDDoQps-smzUPkJd5ru9EHfKuAGRReU2-J-C35kvA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.