Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can
only restart services of a plugin, i.e. lifecycle components with carefully
implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions, parsers,
functions etc. as modules by the onModule...() mechanism, so they get
"baked" into the ES node as immutable objects before the node is going up.
This means, the deploy plugin can not change or undo this behavior. (Maybe
some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web server
that does not use baked-in ES modules, just a restartable service. It is
really delightful to just use a single curl PUT command for rapid
prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:

Jörg

On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe radu.gheorghe@sematext.com
wrote:

Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do
a rolling restart (sticking to 1.3.2), then do the upgrade with another
rolling restart, then install the plugin back again with yet another
rolling restart?

I would understand if you said "no way I do 3 restarts!" :slight_smile: But maybe this
will help in future:
GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins

Best regards,
Radu

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg benberg86@gmail.com wrote:

Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I
upgraded the first node,removed and reinstalled latest versions of plugins

  • two non-site plugins are river-twitter (version 2.4.1) and jdbc (version
    1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the
    errors below in logs on upgraded node and node does not join cluster. If i
    downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin
    version 1.3.0.4 the node properly joins the cluster.
    Errors from logs:
    [2015-01-06 21:29:39,970][INFO ][node ] [bi-es1]
    version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
    [2015-01-06 21:29:39,971][INFO ][node ] [bi-es1]
    initializing ...
    [2015-01-06 21:29:40,009][INFO ][plugins ] [bi-es1]
    loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk,
    paramedic, HQ, whatson]
    [2015-01-06 21:29:45,587][INFO ][node ] [bi-es1]
    initialized
    [2015-01-06 21:29:45,588][INFO ][node ] [bi-es1]
    starting ...
    [2015-01-06 21:29:45,960][INFO ][transport ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
    192.168.83.231:9300]}
    [2015-01-06 21:29:45,982][INFO ][discovery ] [bi-es1]
    bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
    [2015-01-06 21:29:51,906][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28805963] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:51,915][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:29:57,028][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28807127] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:57,036][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:02,245][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28808254] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:02,252][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:07,576][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28809377] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:07,583][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:12,689][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28810050] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:12,696][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:15,989][WARN ][discovery ] [bi-es1]
    waited for 30s and no initial state was set by the discovery
    [2015-01-06 21:30:16,002][INFO ][http ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
    192.168.83.231:9200]}
    [2015-01-06 21:30:16,003][INFO ][node ] [bi-es1]
    started
    [2015-01-06 21:30:18,277][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28811164] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:18,292][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no
    known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results
in same errors. I have tried closing the _river index and removing all
defined rivers (wasn't hopeful that would work but gave it a shot).

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem
good except still on 1.3.2. Would prefer not to do a full cluster shutdown
to upgrade all nodes and jdbc plugin but that seems maybe it would solve
this?

I have had issues in past with upgrades/jdbc river plugin (really love
the plugin though) so my brain is focusing on that and might not be the
issue? I used same process on a test cluster that I just realized did not
have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.