Upgrade client and server from 0.16.x to 0.17.x

Elastic Stack Elasticsearch
1

I have a client 0.16.1 (patched with some bug fix), and a server in 0.16.1.
I am starting to plan to upgrade to 0.17.4. But it seems the "zen" protocol is not backward compatible between the 0.16 and the 0.17. Trying to upgrade on and not the other, I got errors.

Trying the client 0.17 and the server 0.16, I get on the client side:

11.08.2011 15:21:35 WARN [org.elasticsearch.transport.netty:86] [Arsenal] Message not fully read (response) for [0] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@1fb93cf8), error [false], resetting
11.08.2011 15:21:35 INFO [org.elasticsearch.discovery.zen:78] [Arsenal] failed to send join request to master [[Fury, Jacob "Jake"][aaEPP6yGQs6ooO4-1iq5ig][inet[/10.0.0.238:9301]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; java.io.IOException: Expected handle header, got [111]]
11.08.2011 15:21:38 INFO [org.elasticsearch.discovery.zen:78] [Arsenal] failed to send join request to master [[Fury, Jacob "Jake"][aaEPP6yGQs6ooO4-1iq5ig][inet[/10.0.0.238:9301]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; java.io.IOException: Expected handle header, got [111]]
11.08.2011 15:21:38 WARN [org.elasticsearch.transport.netty:86] [Arsenal] Message not fully read (response) for [1] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@2eb6dfa8), error [false], resetting
11.08.2011 15:21:41 WARN [org.elasticsearch.transport.netty:86] [Arsenal] Message not fully read (response) for [2] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@2bf06804), error [false], resetting

On the elasticsearch server side:

[2011-08-11 15:21:35,432][INFO ][cluster.service ] [Fury, Jacob "Jake"] added {[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false}])
[2011-08-11 15:21:38,442][WARN ][discovery.zen ] [Fury, Jacob "Jake"] received a join request for an existing node [[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false}]
[2011-08-11 15:21:41,448][WARN ][discovery.zen ] [Fury, Jacob "Jake"] received a join request for an existing node [[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false}]
[2011-08-11 15:21:43,031][INFO ][cluster.service ] [Fury, Jacob "Jake"] removed {[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false},}, reason: zen-disco-node_failed([Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false}), reason transport disconnected (with verified connect)

With the client 0.16 and the server 0.17, I get this on the client side:

11.08.2011 15:36:53 INFO [org.elasticsearch.discovery.zen:78] [Green Goblin] failed to send join request to master [[Alistaire Stuart][LAcuQv8rTyaxJJ-aDG2v5Q][inet[/10.0.0.238:9300]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; java.io.EOFException]

On the server side:

[2011-08-11 15:36:56,325][e[33mWARN e[0m][discovery.zen ] [Alistaire Stuart] received a join request for an existing node [[Green Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true, data=false}]
[2011-08-11 15:36:57,742][e[32mINFO e[0m][cluster.service ] [Alistaire Stuart] removed {[Green Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([Green Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true, data=false}), reason transport disconnected (with verified connect)

So to do the upgrade, I must have the entire cluster down, my web frontends (clients) and the elasticsearch servers ?

Nicolas

2

Yes.

2011/8/11 Nicolas Lalevée nicolas.lalevee@hibnet.org

I have a client 0.16.1 (patched with some bug fix), and a server in 0.16.1.
I am starting to plan to upgrade to 0.17.4. But it seems the "zen" protocol
is not backward compatible between the 0.16 and the 0.17. Trying to upgrade
on and not the other, I got errors.

Trying the client 0.17 and the server 0.16, I get on the client side:

11.08.2011 15:21:35 WARN [org.elasticsearch.transport.netty:86] [Arsenal]
Message not fully read (response) for [0] handler
future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@1fb93cf8),
error [false], resetting
11.08.2011 15:21:35 INFO [org.elasticsearch.discovery.zen:78] [Arsenal]
failed to send join request to master [[Fury, Jacob
"Jake"][aaEPP6yGQs6ooO4-1iq5ig][inet[/10.0.0.238:9301]]], reason
[org.elasticsearch.transport.RemoteTransportException: Failed to deserialize
response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
org.elasticsearch.transport.TransportSerializationException: Failed to
deserialize response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
java.io.IOException: Expected handle header, got [111]]
11.08.2011 15:21:38 INFO [org.elasticsearch.discovery.zen:78] [Arsenal]
failed to send join request to master [[Fury, Jacob
"Jake"][aaEPP6yGQs6ooO4-1iq5ig][inet[/10.0.0.238:9301]]], reason
[org.elasticsearch.transport.RemoteTransportException: Failed to deserialize
response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
org.elasticsearch.transport.TransportSerializationException: Failed to
deserialize response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
java.io.IOException: Expected handle header, got [111]]
11.08.2011 15:21:38 WARN [org.elasticsearch.transport.netty:86] [Arsenal]
Message not fully read (response) for [1] handler
future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@2eb6dfa8),
error [false], resetting
11.08.2011 15:21:41 WARN [org.elasticsearch.transport.netty:86] [Arsenal]
Message not fully read (response) for [2] handler
future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@2bf06804),
error [false], resetting

On the elasticsearch server side:

[2011-08-11 15:21:35,432][INFO ][cluster.service ] [Fury, Jacob
"Jake"] added {[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false}])
[2011-08-11 15:21:38,442][WARN ][discovery.zen ] [Fury, Jacob
"Jake"] received a join request for an existing node
[[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false}]
[2011-08-11 15:21:41,448][WARN ][discovery.zen ] [Fury, Jacob
"Jake"] received a join request for an existing node
[[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false}]
[2011-08-11 15:21:43,031][INFO ][cluster.service ] [Fury, Jacob
"Jake"] removed {[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false},}, reason:
zen-disco-node_failed([Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false}), reason transport disconnected (with verified connect)

With the client 0.16 and the server 0.17, I get this on the client side:

11.08.2011 15:36:53 INFO [org.elasticsearch.discovery.zen:78] [Green
Goblin] failed to send join request to master [[Alistaire
Stuart][LAcuQv8rTyaxJJ-aDG2v5Q][inet[/10.0.0.238:9300]]], reason
[org.elasticsearch.transport.RemoteTransportException: Failed to deserialize
response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
org.elasticsearch.transport.TransportSerializationException: Failed to
deserialize response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
java.io.EOFException]

On the server side:

[2011-08-11 15:36:56,325][ [33mWARN [0m][discovery.zen ]
[Alistaire Stuart] received a join request for an existing node [[Green
Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true,
data=false}]
[2011-08-11 15:36:57,742][ [32mINFO [0m][cluster.service ]
[Alistaire Stuart] removed {[Green
Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true,
data=false},}, reason: zen-disco-node_failed([Green
Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

So to do the upgrade, I must have the entire cluster down, my web frontends
(clients) and the elasticsearch servers ?

Nicolas

3

Then it would be great that in the future some backward compatibility is kept. This is somehow part of the feature "search server always available".

cheers,
Nicolas

Le 11 août 2011 à 17:09, Shay Banon a écrit :

Yes.

2011/8/11 Nicolas Lalevée nicolas.lalevee@hibnet.org
I have a client 0.16.1 (patched with some bug fix), and a server in 0.16.1.
I am starting to plan to upgrade to 0.17.4. But it seems the "zen" protocol is not backward compatible between the 0.16 and the 0.17. Trying to upgrade on and not the other, I got errors.

Trying the client 0.17 and the server 0.16, I get on the client side:

11.08.2011 15:21:35 WARN [org.elasticsearch.transport.netty:86] [Arsenal] Message not fully read (response) for [0] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@1fb93cf8), error [false], resetting
11.08.2011 15:21:35 INFO [org.elasticsearch.discovery.zen:78] [Arsenal] failed to send join request to master [[Fury, Jacob "Jake"][aaEPP6yGQs6ooO4-1iq5ig][inet[/10.0.0.238:9301]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; java.io.IOException: Expected handle header, got [111]]
11.08.2011 15:21:38 INFO [org.elasticsearch.discovery.zen:78] [Arsenal] failed to send join request to master [[Fury, Jacob "Jake"][aaEPP6yGQs6ooO4-1iq5ig][inet[/10.0.0.238:9301]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; java.io.IOException: Expected handle header, got [111]]
11.08.2011 15:21:38 WARN [org.elasticsearch.transport.netty:86] [Arsenal] Message not fully read (response) for [1] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@2eb6dfa8), error [false], resetting
11.08.2011 15:21:41 WARN [org.elasticsearch.transport.netty:86] [Arsenal] Message not fully read (response) for [2] handler future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@2bf06804), error [false], resetting

On the elasticsearch server side:

[2011-08-11 15:21:35,432][INFO ][cluster.service ] [Fury, Jacob "Jake"] added {[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false}])
[2011-08-11 15:21:38,442][WARN ][discovery.zen ] [Fury, Jacob "Jake"] received a join request for an existing node [[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false}]
[2011-08-11 15:21:41,448][WARN ][discovery.zen ] [Fury, Jacob "Jake"] received a join request for an existing node [[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false}]
[2011-08-11 15:21:43,031][INFO ][cluster.service ] [Fury, Jacob "Jake"] removed {[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false},}, reason: zen-disco-node_failed([Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true, data=false}), reason transport disconnected (with verified connect)

With the client 0.16 and the server 0.17, I get this on the client side:

11.08.2011 15:36:53 INFO [org.elasticsearch.discovery.zen:78] [Green Goblin] failed to send join request to master [[Alistaire Stuart][LAcuQv8rTyaxJJ-aDG2v5Q][inet[/10.0.0.238:9300]]], reason [org.elasticsearch.transport.RemoteTransportException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; org.elasticsearch.transport.TransportSerializationException: Failed to deserialize response of type [org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse]; java.io.EOFException]

On the server side:

[2011-08-11 15:36:56,325][ [33mWARN [0m][discovery.zen ] [Alistaire Stuart] received a join request for an existing node [[Green Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true, data=false}]
[2011-08-11 15:36:57,742][ [32mINFO [0m][cluster.service ] [Alistaire Stuart] removed {[Green Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([Green Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true, data=false}), reason transport disconnected (with verified connect)

So to do the upgrade, I must have the entire cluster down, my web frontends (clients) and the elasticsearch servers ?

Nicolas

4

Yep, it would be nice.

2011/8/11 Nicolas Lalevée nicolas.lalevee@hibnet.org

Then it would be great that in the future some backward compatibility is
kept. This is somehow part of the feature "search server always available".

cheers,
Nicolas

Le 11 août 2011 à 17:09, Shay Banon a écrit :

Yes.

2011/8/11 Nicolas Lalevée nicolas.lalevee@hibnet.org
I have a client 0.16.1 (patched with some bug fix), and a server in
0.16.1.
I am starting to plan to upgrade to 0.17.4. But it seems the "zen"
protocol is not backward compatible between the 0.16 and the 0.17. Trying to
upgrade on and not the other, I got errors.

Trying the client 0.17 and the server 0.16, I get on the client side:

11.08.2011 15:21:35 WARN [org.elasticsearch.transport.netty:86] [Arsenal]
Message not fully read (response) for [0] handler
future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@1fb93cf8),
error [false], resetting
11.08.2011 15:21:35 INFO [org.elasticsearch.discovery.zen:78] [Arsenal]
failed to send join request to master [[Fury, Jacob
"Jake"][aaEPP6yGQs6ooO4-1iq5ig][inet[/10.0.0.238:9301]]], reason
[org.elasticsearch.transport.RemoteTransportException: Failed to deserialize
response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
org.elasticsearch.transport.TransportSerializationException: Failed to
deserialize response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
java.io.IOException: Expected handle header, got [111]]
11.08.2011 15:21:38 INFO [org.elasticsearch.discovery.zen:78] [Arsenal]
failed to send join request to master [[Fury, Jacob
"Jake"][aaEPP6yGQs6ooO4-1iq5ig][inet[/10.0.0.238:9301]]], reason
[org.elasticsearch.transport.RemoteTransportException: Failed to deserialize
response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
org.elasticsearch.transport.TransportSerializationException: Failed to
deserialize response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
java.io.IOException: Expected handle header, got [111]]
11.08.2011 15:21:38 WARN [org.elasticsearch.transport.netty:86] [Arsenal]
Message not fully read (response) for [1] handler
future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@2eb6dfa8),
error [false], resetting
11.08.2011 15:21:41 WARN [org.elasticsearch.transport.netty:86] [Arsenal]
Message not fully read (response) for [2] handler
future(org.elasticsearch.discovery.zen.membership.MembershipAction$1@2bf06804),
error [false], resetting

On the elasticsearch server side:

[2011-08-11 15:21:35,432][INFO ][cluster.service ] [Fury, Jacob
"Jake"] added {[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false},}, reason: zen-disco-receive(join from
node[[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false}])
[2011-08-11 15:21:38,442][WARN ][discovery.zen ] [Fury, Jacob
"Jake"] received a join request for an existing node
[[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false}]
[2011-08-11 15:21:41,448][WARN ][discovery.zen ] [Fury, Jacob
"Jake"] received a join request for an existing node
[[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false}]
[2011-08-11 15:21:43,031][INFO ][cluster.service ] [Fury, Jacob
"Jake"] removed {[Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false},}, reason:
zen-disco-node_failed([Arsenal][wdaOfDovR-WZr91kCR9Pmw][inet[/10.0.0.238:9300]]{client=true,
data=false}), reason transport disconnected (with verified connect)

With the client 0.16 and the server 0.17, I get this on the client side:

11.08.2011 15:36:53 INFO [org.elasticsearch.discovery.zen:78] [Green
Goblin] failed to send join request to master [[Alistaire
Stuart][LAcuQv8rTyaxJJ-aDG2v5Q][inet[/10.0.0.238:9300]]], reason
[org.elasticsearch.transport.RemoteTransportException: Failed to deserialize
response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
org.elasticsearch.transport.TransportSerializationException: Failed to
deserialize response of type
[org.elasticsearch.discovery.zen.membership.MembershipAction$JoinResponse];
java.io.EOFException]

On the server side:

[2011-08-11 15:36:56,325][ [33mWARN [0m][discovery.zen ]
[Alistaire Stuart] received a join request for an existing node [[Green
Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true,
data=false}]
[2011-08-11 15:36:57,742][ [32mINFO [0m][cluster.service ]
[Alistaire Stuart] removed {[Green
Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true,
data=false},}, reason: zen-disco-node_failed([Green
Goblin][fNxusRThSRqyOnsCYyFohQ][inet[/10.0.0.238:9301]]{client=true,
data=false}), reason transport disconnected (with verified connect)

So to do the upgrade, I must have the entire cluster down, my web
frontends (clients) and the elasticsearch servers ?

Nicolas

5

I am starting to plan to upgrade to 0.17.4

Could you create a replica, break it off and start that as a separate
cluster, upgrade the new cluster, run it in read-only mode (so you can
service client requests but do not need to synchronize data).

Now you have time to update the main cluster because clients are running off
the temporary, new cluster. This assumes the hardware requirements are
feasible, and no indexing is required during this time.

6

Le 11 août 2011 à 19:53, Tom Le a écrit :

I am starting to plan to upgrade to 0.17.4

Could you create a replica, break it off and start that as a separate cluster, upgrade the new cluster, run it in read-only mode (so you can service client requests but do not need to synchronize data).

Now you have time to update the main cluster because clients are running off the temporary, new cluster. This assumes the hardware requirements are feasible, and no indexing is required during this time.

Not having write availability is like having a downtime for the product I develop. So it will be simpler to get everything down.
And we are not that obsessed with availability since we authorize ourself to have downtime due to some sql schema migration. But we try hard to not have downtime if its not absolutely required.

Nicolas

7

Agreed, the aim for elasticsearch is to get to a state where it will have
backward and forward compatible communication channels. But thats down the
road...

2011/8/11 Nicolas Lalevée nicolas.lalevee@hibnet.org

Le 11 août 2011 à 19:53, Tom Le a écrit :

I am starting to plan to upgrade to 0.17.4

Could you create a replica, break it off and start that as a separate
cluster, upgrade the new cluster, run it in read-only mode (so you can
service client requests but do not need to synchronize data).

Now you have time to update the main cluster because clients are running
off the temporary, new cluster. This assumes the hardware requirements are
feasible, and no indexing is required during this time.

Not having write availability is like having a downtime for the product I
develop. So it will be simpler to get everything down.
And we are not that obsessed with availability since we authorize ourself
to have downtime due to some sql schema migration. But we try hard to not
have downtime if its not absolutely required.

Nicolas