Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2

Elastic Stack Elasticsearch
1

Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I
upgraded the first node,removed and reinstalled latest versions of plugins

  • two non-site plugins are river-twitter (version 2.4.1) and jdbc (version
    1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the
    errors below in logs on upgraded node and node does not join cluster. If i
    downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin
    version 1.3.0.4 the node properly joins the cluster.
    Errors from logs:
    [2015-01-06 21:29:39,970][INFO ][node ] [bi-es1]
    version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
    [2015-01-06 21:29:39,971][INFO ][node ] [bi-es1]
    initializing ...
    [2015-01-06 21:29:40,009][INFO ][plugins ] [bi-es1] loaded
    [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk,
    paramedic, HQ, whatson]
    [2015-01-06 21:29:45,587][INFO ][node ] [bi-es1]
    initialized
    [2015-01-06 21:29:45,588][INFO ][node ] [bi-es1]
    starting ...
    [2015-01-06 21:29:45,960][INFO ][transport ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address
    {inet[/192.168.83.231:9300]}
    [2015-01-06 21:29:45,982][INFO ][discovery ] [bi-es1]
    bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
    [2015-01-06 21:29:51,906][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28805963] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:51,915][INFO ][discovery.zen ] [bi-es1] failed
    to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason
    [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested:
    RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:29:57,028][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28807127] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:57,036][INFO ][discovery.zen ] [bi-es1] failed
    to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason
    [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested:
    RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:02,245][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28808254] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:02,252][INFO ][discovery.zen ] [bi-es1] failed
    to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason
    [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested:
    RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:07,576][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28809377] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:07,583][INFO ][discovery.zen ] [bi-es1] failed
    to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason
    [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested:
    RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:12,689][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28810050] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:12,696][INFO ][discovery.zen ] [bi-es1] failed
    to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason
    [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested:
    RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:15,989][WARN ][discovery ] [bi-es1] waited
    for 30s and no initial state was set by the discovery
    [2015-01-06 21:30:16,002][INFO ][http ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address
    {inet[/192.168.83.231:9200]}
    [2015-01-06 21:30:16,003][INFO ][node ] [bi-es1] started
    [2015-01-06 21:30:18,277][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28811164] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:18,292][INFO ][discovery.zen ] [bi-es1] failed
    to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason
    [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested:
    RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no
    known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results
in same errors. I have tried closing the _river index and removing all
defined rivers (wasn't hopeful that would work but gave it a shot).

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem good
except still on 1.3.2. Would prefer not to do a full cluster shutdown to
upgrade all nodes and jdbc plugin but that seems maybe it would solve this?

I have had issues in past with upgrades/jdbc river plugin (really love the
plugin though) so my brain is focusing on that and might not be the issue?
I used same process on a test cluster that I just realized did not have the
jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do a
rolling restart (sticking to 1.3.2), then do the upgrade with another
rolling restart, then install the plugin back again with yet another
rolling restart?

I would understand if you said "no way I do 3 restarts!" :slight_smile: But maybe this
will help in future: GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins

Best regards,
Radu

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg benberg86@gmail.com wrote:

Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I
upgraded the first node,removed and reinstalled latest versions of plugins

  • two non-site plugins are river-twitter (version 2.4.1) and jdbc (version
    1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the
    errors below in logs on upgraded node and node does not join cluster. If i
    downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin
    version 1.3.0.4 the node properly joins the cluster.
    Errors from logs:
    [2015-01-06 21:29:39,970][INFO ][node ] [bi-es1]
    version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
    [2015-01-06 21:29:39,971][INFO ][node ] [bi-es1]
    initializing ...
    [2015-01-06 21:29:40,009][INFO ][plugins ] [bi-es1]
    loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk,
    paramedic, HQ, whatson]
    [2015-01-06 21:29:45,587][INFO ][node ] [bi-es1]
    initialized
    [2015-01-06 21:29:45,588][INFO ][node ] [bi-es1]
    starting ...
    [2015-01-06 21:29:45,960][INFO ][transport ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
    192.168.83.231:9300]}
    [2015-01-06 21:29:45,982][INFO ][discovery ] [bi-es1]
    bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
    [2015-01-06 21:29:51,906][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28805963] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:51,915][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:29:57,028][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28807127] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:57,036][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:02,245][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28808254] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:02,252][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:07,576][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28809377] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:07,583][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:12,689][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28810050] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:12,696][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:15,989][WARN ][discovery ] [bi-es1]
    waited for 30s and no initial state was set by the discovery
    [2015-01-06 21:30:16,002][INFO ][http ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
    192.168.83.231:9200]}
    [2015-01-06 21:30:16,003][INFO ][node ] [bi-es1]
    started
    [2015-01-06 21:30:18,277][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28811164] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:18,292][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no
    known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results
in same errors. I have tried closing the _river index and removing all
defined rivers (wasn't hopeful that would work but gave it a shot).

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem
good except still on 1.3.2. Would prefer not to do a full cluster shutdown
to upgrade all nodes and jdbc plugin but that seems maybe it would solve
this?

I have had issues in past with upgrades/jdbc river plugin (really love the
plugin though) so my brain is focusing on that and might not be the issue?
I used same process on a test cluster that I just realized did not have the
jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks for the reply!

That is a good idea to try, but another problem with this is that when i
uninstall the plugins and keep at version 1.3.2 the node will not join
cluster either.
We upgraded to java 8 right before starting this upgrade and thinking it
may have something to do with that - also noticed that somehow JAVA_HOME
was not defined - working on setting that up across the cluster and will
attempt to remove plugin while on 1.3.2 and get node to join cluster - if
that does not work I will downgrade to java 7 and try again.
Thanks again for reply, insights and huge thanks for link to the plugin
repo - looks awesome and will probably try it after sorting this out. Will
update with findings later today hopefully.
Ben

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b5dfdbb1-507e-4032-9ec2-40d91266432b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Ended up finding out that the jdbc plugin requires a full cluster shutdown
for restart - https://github.com/jprante/elasticsearch-river-jdbc/issues/433
Going to be doing that in maintenance window tomorrow and will verify that
worked.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/60577717-944f-4b13-b66d-124cfb2e7b39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can
only restart services of a plugin, i.e. lifecycle components with carefully
implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions, parsers,
functions etc. as modules by the onModule...() mechanism, so they get
"baked" into the ES node as immutable objects before the node is going up.
This means, the deploy plugin can not change or undo this behavior. (Maybe
some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web server
that does not use baked-in ES modules, just a restartable service. It is
really delightful to just use a single curl PUT command for rapid
prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:

Jörg

On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe radu.gheorghe@sematext.com
wrote:

Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do
a rolling restart (sticking to 1.3.2), then do the upgrade with another
rolling restart, then install the plugin back again with yet another
rolling restart?

I would understand if you said "no way I do 3 restarts!" :slight_smile: But maybe this
will help in future:
GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins

Best regards,
Radu

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg benberg86@gmail.com wrote:

Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I
upgraded the first node,removed and reinstalled latest versions of plugins

  • two non-site plugins are river-twitter (version 2.4.1) and jdbc (version
    1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the
    errors below in logs on upgraded node and node does not join cluster. If i
    downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin
    version 1.3.0.4 the node properly joins the cluster.
    Errors from logs:
    [2015-01-06 21:29:39,970][INFO ][node ] [bi-es1]
    version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
    [2015-01-06 21:29:39,971][INFO ][node ] [bi-es1]
    initializing ...
    [2015-01-06 21:29:40,009][INFO ][plugins ] [bi-es1]
    loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk,
    paramedic, HQ, whatson]
    [2015-01-06 21:29:45,587][INFO ][node ] [bi-es1]
    initialized
    [2015-01-06 21:29:45,588][INFO ][node ] [bi-es1]
    starting ...
    [2015-01-06 21:29:45,960][INFO ][transport ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
    192.168.83.231:9300]}
    [2015-01-06 21:29:45,982][INFO ][discovery ] [bi-es1]
    bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
    [2015-01-06 21:29:51,906][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28805963] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:51,915][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:29:57,028][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28807127] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:57,036][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:02,245][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28808254] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:02,252][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:07,576][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28809377] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:07,583][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:12,689][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28810050] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:12,696][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:15,989][WARN ][discovery ] [bi-es1]
    waited for 30s and no initial state was set by the discovery
    [2015-01-06 21:30:16,002][INFO ][http ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
    192.168.83.231:9200]}
    [2015-01-06 21:30:16,003][INFO ][node ] [bi-es1]
    started
    [2015-01-06 21:30:18,277][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28811164] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:18,292][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no
    known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that results
in same errors. I have tried closing the _river index and removing all
defined rivers (wasn't hopeful that would work but gave it a shot).

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem
good except still on 1.3.2. Would prefer not to do a full cluster shutdown
to upgrade all nodes and jdbc plugin but that seems maybe it would solve
this?

I have had issues in past with upgrades/jdbc river plugin (really love
the plugin though) so my brain is focusing on that and might not be the
issue? I used same process on a test cluster that I just realized did not
have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

6

Thanks Ben and Joerg. Too bad there was no good solution this particular
upgrade, at least we all learned something (well, aside from Joerg who knew
it all from the start :D).

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 12:28 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can
only restart services of a plugin, i.e. lifecycle components with carefully
implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions, parsers,
functions etc. as modules by the onModule...() mechanism, so they get
"baked" into the ES node as immutable objects before the node is going up.
This means, the deploy plugin can not change or undo this behavior. (Maybe
some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web server
that does not use baked-in ES modules, just a restartable service. It is
really delightful to just use a single curl PUT command for rapid
prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:

GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for embedding Ratpack http://ratpack.io

Jörg

On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe radu.gheorghe@sematext.com
wrote:

Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and do
a rolling restart (sticking to 1.3.2), then do the upgrade with another
rolling restart, then install the plugin back again with yet another
rolling restart?

I would understand if you said "no way I do 3 restarts!" :slight_smile: But maybe
this will help in future:
GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins

Best regards,
Radu

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg benberg86@gmail.com wrote:

Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I
upgraded the first node,removed and reinstalled latest versions of plugins

  • two non-site plugins are river-twitter (version 2.4.1) and jdbc (version
    1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the
    errors below in logs on upgraded node and node does not join cluster. If i
    downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin
    version 1.3.0.4 the node properly joins the cluster.
    Errors from logs:
    [2015-01-06 21:29:39,970][INFO ][node ] [bi-es1]
    version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
    [2015-01-06 21:29:39,971][INFO ][node ] [bi-es1]
    initializing ...
    [2015-01-06 21:29:40,009][INFO ][plugins ] [bi-es1]
    loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk,
    paramedic, HQ, whatson]
    [2015-01-06 21:29:45,587][INFO ][node ] [bi-es1]
    initialized
    [2015-01-06 21:29:45,588][INFO ][node ] [bi-es1]
    starting ...
    [2015-01-06 21:29:45,960][INFO ][transport ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
    192.168.83.231:9300]}
    [2015-01-06 21:29:45,982][INFO ][discovery ] [bi-es1]
    bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
    [2015-01-06 21:29:51,906][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28805963] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:51,915][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:29:57,028][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28807127] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:57,036][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:02,245][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28808254] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:02,252][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:07,576][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28809377] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:07,583][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:12,689][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28810050] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:12,696][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:15,989][WARN ][discovery ] [bi-es1]
    waited for 30s and no initial state was set by the discovery
    [2015-01-06 21:30:16,002][INFO ][http ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
    192.168.83.231:9200]}
    [2015-01-06 21:30:16,003][INFO ][node ] [bi-es1]
    started
    [2015-01-06 21:30:18,277][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28811164] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:18,292][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no
    known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that
results in same errors. I have tried closing the _river index and removing
all defined rivers (wasn't hopeful that would work but gave it a shot).

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem
good except still on 1.3.2. Would prefer not to do a full cluster shutdown
to upgrade all nodes and jdbc plugin but that seems maybe it would solve
this?

I have had issues in past with upgrades/jdbc river plugin (really love
the plugin though) so my brain is focusing on that and might not be the
issue? I used same process on a test cluster that I just realized did not
have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

No I didn't :frowning:

I still struggle with stateful plugins - there is no good solution I know
of and that is the reason of trouble.

Jörg

On Fri, Jan 9, 2015 at 9:01 AM, Radu Gheorghe radu.gheorghe@sematext.com
wrote:

Thanks Ben and Joerg. Too bad there was no good solution this particular
upgrade, at least we all learned something (well, aside from Joerg who knew
it all from the start :D).

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 12:28 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can
only restart services of a plugin, i.e. lifecycle components with carefully
implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions, parsers,
functions etc. as modules by the onModule...() mechanism, so they get
"baked" into the ES node as immutable objects before the node is going up.
This means, the deploy plugin can not change or undo this behavior. (Maybe
some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web
server that does not use baked-in ES modules, just a restartable service.
It is really delightful to just use a single curl PUT command for rapid
prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:

GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for embedding Ratpack http://ratpack.io

Jörg

On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe <radu.gheorghe@sematext.com

wrote:

Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and
do a rolling restart (sticking to 1.3.2), then do the upgrade with another
rolling restart, then install the plugin back again with yet another
rolling restart?

I would understand if you said "no way I do 3 restarts!" :slight_smile: But maybe
this will help in future:
GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins

Best regards,
Radu

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg benberg86@gmail.com wrote:

Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I
upgraded the first node,removed and reinstalled latest versions of plugins

  • two non-site plugins are river-twitter (version 2.4.1) and jdbc (version
    1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the
    errors below in logs on upgraded node and node does not join cluster. If i
    downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin
    version 1.3.0.4 the node properly joins the cluster.
    Errors from logs:
    [2015-01-06 21:29:39,970][INFO ][node ] [bi-es1]
    version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
    [2015-01-06 21:29:39,971][INFO ][node ] [bi-es1]
    initializing ...
    [2015-01-06 21:29:40,009][INFO ][plugins ] [bi-es1]
    loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk,
    paramedic, HQ, whatson]
    [2015-01-06 21:29:45,587][INFO ][node ] [bi-es1]
    initialized
    [2015-01-06 21:29:45,588][INFO ][node ] [bi-es1]
    starting ...
    [2015-01-06 21:29:45,960][INFO ][transport ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
    192.168.83.231:9300]}
    [2015-01-06 21:29:45,982][INFO ][discovery ] [bi-es1]
    bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
    [2015-01-06 21:29:51,906][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28805963] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:51,915][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:29:57,028][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28807127] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:57,036][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:02,245][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28808254] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:02,252][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:07,576][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28809377] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:07,583][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:12,689][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28810050] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:12,696][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:15,989][WARN ][discovery ] [bi-es1]
    waited for 30s and no initial state was set by the discovery
    [2015-01-06 21:30:16,002][INFO ][http ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
    192.168.83.231:9200]}
    [2015-01-06 21:30:16,003][INFO ][node ] [bi-es1]
    started
    [2015-01-06 21:30:18,277][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28811164] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:18,292][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1]
    no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that
results in same errors. I have tried closing the _river index and removing
all defined rivers (wasn't hopeful that would work but gave it a shot).

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem
good except still on 1.3.2. Would prefer not to do a full cluster shutdown
to upgrade all nodes and jdbc plugin but that seems maybe it would solve
this?

I have had issues in past with upgrades/jdbc river plugin (really love
the plugin though) so my brain is focusing on that and might not be the
issue? I used same process on a test cluster that I just realized did not
have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

8

Yeah, my general approach so far was to stay away from plugins as much as
possible and use external services instead. Something that your JDBC river
can do as well, as far as I understand.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 10:20 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

No I didn't :frowning:

I still struggle with stateful plugins - there is no good solution I know
of and that is the reason of trouble.

Jörg

On Fri, Jan 9, 2015 at 9:01 AM, Radu Gheorghe radu.gheorghe@sematext.com
wrote:

Thanks Ben and Joerg. Too bad there was no good solution this particular
upgrade, at least we all learned something (well, aside from Joerg who knew
it all from the start :D).

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 12:28 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it can
only restart services of a plugin, i.e. lifecycle components with carefully
implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions,
parsers, functions etc. as modules by the onModule...() mechanism, so they
get "baked" into the ES node as immutable objects before the node is going
up. This means, the deploy plugin can not change or undo this behavior.
(Maybe some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web
server that does not use baked-in ES modules, just a restartable service.
It is really delightful to just use a single curl PUT command for rapid
prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:

GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for embedding Ratpack http://ratpack.io

Jörg

On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe <
radu.gheorghe@sematext.com> wrote:

Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and
do a rolling restart (sticking to 1.3.2), then do the upgrade with another
rolling restart, then install the plugin back again with yet another
rolling restart?

I would understand if you said "no way I do 3 restarts!" :slight_smile: But maybe
this will help in future:
GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins

Best regards,
Radu

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg benberg86@gmail.com wrote:

Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I
upgraded the first node,removed and reinstalled latest versions of plugins

  • two non-site plugins are river-twitter (version 2.4.1) and jdbc (version
    1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the
    errors below in logs on upgraded node and node does not join cluster. If i
    downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin
    version 1.3.0.4 the node properly joins the cluster.
    Errors from logs:
    [2015-01-06 21:29:39,970][INFO ][node ] [bi-es1]
    version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
    [2015-01-06 21:29:39,971][INFO ][node ] [bi-es1]
    initializing ...
    [2015-01-06 21:29:40,009][INFO ][plugins ] [bi-es1]
    loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk,
    paramedic, HQ, whatson]
    [2015-01-06 21:29:45,587][INFO ][node ] [bi-es1]
    initialized
    [2015-01-06 21:29:45,588][INFO ][node ] [bi-es1]
    starting ...
    [2015-01-06 21:29:45,960][INFO ][transport ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
    192.168.83.231:9300]}
    [2015-01-06 21:29:45,982][INFO ][discovery ] [bi-es1]
    bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
    [2015-01-06 21:29:51,906][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28805963] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:51,915][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:29:57,028][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28807127] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:57,036][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:02,245][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28808254] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:02,252][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:07,576][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28809377] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:07,583][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:12,689][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28810050] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:12,696][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:15,989][WARN ][discovery ] [bi-es1]
    waited for 30s and no initial state was set by the discovery
    [2015-01-06 21:30:16,002][INFO ][http ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
    192.168.83.231:9200]}
    [2015-01-06 21:30:16,003][INFO ][node ] [bi-es1]
    started
    [2015-01-06 21:30:18,277][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28811164] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:18,292][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1]
    no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that
results in same errors. I have tried closing the _river index and removing
all defined rivers (wasn't hopeful that would work but gave it a shot).

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things seem
good except still on 1.3.2. Would prefer not to do a full cluster shutdown
to upgrade all nodes and jdbc plugin but that seems maybe it would solve
this?

I have had issues in past with upgrades/jdbc river plugin (really love
the plugin though) so my brain is focusing on that and might not be the
issue? I used same process on a test cluster that I just realized did not
have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

9

I could switch away from plugin mode, but that would mean people must adapt
and sell another JDBC bridge component to their boss and worse, that will
keep the SPOF weakness of rivers but with more burden of responsibility on
myself. Not in my sparetime.

Maybe a compromise will work better: external service, but also attached to
the ES nodes as a "lightweight" restartable plugin. It looks attractive to
me to move over to Groovy and run SQL / JDBC from there. The picture is an
enhanced deploy plugin and a general harvester job distribution module with
UI, where JDBC is just one option. With the deploy plugin it is easy to
move JDBC driver jars and other dependencies around in the cluster, for
stop/start, keep state etc.

Jörg

On Fri, Jan 9, 2015 at 9:22 AM, Radu Gheorghe radu.gheorghe@sematext.com
wrote:

Yeah, my general approach so far was to stay away from plugins as much as
possible and use external services instead. Something that your JDBC river
can do as well, as far as I understand.

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 10:20 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

No I didn't :frowning:

I still struggle with stateful plugins - there is no good solution I know
of and that is the reason of trouble.

Jörg

On Fri, Jan 9, 2015 at 9:01 AM, Radu Gheorghe <radu.gheorghe@sematext.com

wrote:

Thanks Ben and Joerg. Too bad there was no good solution this particular
upgrade, at least we all learned something (well, aside from Joerg who knew
it all from the start :D).

Best regards,
Radu

--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Fri, Jan 9, 2015 at 12:28 AM, joergprante@gmail.com <
joergprante@gmail.com> wrote:

Hi Radu,

the deploy plugin is somewhat limited to "restartable" plugins as it
can only restart services of a plugin, i.e. lifecycle components with
carefully implemented doStart()/doStop() methods.

Most plugins come with modules that add REST endpoints, actions,
parsers, functions etc. as modules by the onModule...() mechanism, so they
get "baked" into the ES node as immutable objects before the node is going
up. This means, the deploy plugin can not change or undo this behavior.
(Maybe some evil hackery will do to a certain degree)

For my use case, I embed a Ratpack server and start my own HTTP web
server that does not use baked-in ES modules, just a restartable service.
It is really delightful to just use a single curl PUT command for rapid
prototyping a Ratpack web app embedded in ES without node restart.

A restartable plugin is:

GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for embedding Ratpack http://ratpack.io

Jörg

On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe <
radu.gheorghe@sematext.com> wrote:

Hello Ben,

Maybe it works if you uninstall the plugin from one node at a time and
do a rolling restart (sticking to 1.3.2), then do the upgrade with another
rolling restart, then install the plugin back again with yet another
rolling restart?

I would understand if you said "no way I do 3 restarts!" :slight_smile: But maybe
this will help in future:
GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins

Best regards,
Radu

Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/

On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg benberg86@gmail.com wrote:

Hello,
I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I
upgraded the first node,removed and reinstalled latest versions of plugins

  • two non-site plugins are river-twitter (version 2.4.1) and jdbc (version
    1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the
    errors below in logs on upgraded node and node does not join cluster. If i
    downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin
    version 1.3.0.4 the node properly joins the cluster.
    Errors from logs:
    [2015-01-06 21:29:39,970][INFO ][node ] [bi-es1]
    version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z]
    [2015-01-06 21:29:39,971][INFO ][node ] [bi-es1]
    initializing ...
    [2015-01-06 21:29:40,009][INFO ][plugins ] [bi-es1]
    loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk,
    paramedic, HQ, whatson]
    [2015-01-06 21:29:45,587][INFO ][node ] [bi-es1]
    initialized
    [2015-01-06 21:29:45,588][INFO ][node ] [bi-es1]
    starting ...
    [2015-01-06 21:29:45,960][INFO ][transport ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/
    192.168.83.231:9300]}
    [2015-01-06 21:29:45,982][INFO ][discovery ] [bi-es1]
    bi-cluster1/VyTzSKWnQAS9ZCk971fJEg
    [2015-01-06 21:29:51,906][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28805963] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:51,915][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:29:57,028][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28807127] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:29:57,036][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:02,245][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28808254] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:02,252][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:07,576][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28809377] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:07,583][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:12,689][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28810050] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:12,696][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:15,989][WARN ][discovery ] [bi-es1]
    waited for 30s and no initial state was set by the discovery
    [2015-01-06 21:30:16,002][INFO ][http ] [bi-es1]
    bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/
    192.168.83.231:9200]}
    [2015-01-06 21:30:16,003][INFO ][node ] [bi-es1]
    started
    [2015-01-06 21:30:18,277][WARN ][transport.netty ] [bi-es1]
    Message not fully read (request) for [28811164] and action
    [discovery/zen/join/validate], resetting
    [2015-01-06 21:30:18,292][INFO ][discovery.zen ] [bi-es1]
    failed to send join request to master
    [[bi-es4][-cczOZfKTw-duMdUO_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}],
    reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]];
    nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]];
    nested: ElasticsearchIllegalArgumentException[No custom index metadata
    factory registered for type [rivers]]; ]
    [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1]
    no known master node, scheduling a retry

I have tried removing all plugins from the upgraded node and that
results in same errors. I have tried closing the _river index and removing
all defined rivers (wasn't hopeful that would work but gave it a shot).

I am not sure what else to try - thoughts? ideas?

I am safely downgraded back to 1.3.2 and node rejoined and things
seem good except still on 1.3.2. Would prefer not to do a full cluster
shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would
solve this?

I have had issues in past with upgrades/jdbc river plugin (really
love the plugin though) so my brain is focusing on that and might not be
the issue? I used same process on a test cluster that I just realized did
not have the jdbc plugin installed on it and that went smooth.

Any help would be greatly appreciated!
Thanks
Ben

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEs%3D8BFHS59eXES0gSwY2Z6M1K9Ctz3J7Z1ZZ8CJk1K7g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.