# Upgraded node unable to join cluster while attempting cluster upgrade from 1.3.2 to 1.4.2 **URL:** \ **Category:** Elasticsearch\ **Created:** [January 7, 2015, 3:15am UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516 "2015-01-07T03:15:40Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Ben\_Berg](https://avatars.discourse-cdn.com/v4/letter/b/8baadc/32.png) [@Ben\_Berg](https://discuss.elastic.co/u/Ben_Berg)\ **Post date:** [January 7, 2015, 3:15am UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/1 "2015-01-07T03:15:40Z")
Hello, I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I upgraded the first node,removed and reinstalled latest versions of plugins - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the errors below in logs on upgraded node and node does not join cluster. If i downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin version 1.3.0.4 the node properly joins the cluster. Errors from logs: [2015-01-06 21:29:39,970][INFO][node] [bi-es1] version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z] [2015-01-06 21:29:39,971][INFO][node] [bi-es1] initializing ... [2015-01-06 21:29:40,009][INFO][plugins] [bi-es1] loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, paramedic, HQ, whatson] [2015-01-06 21:29:45,587][INFO][node] [bi-es1] initialized [2015-01-06 21:29:45,588][INFO][node] [bi-es1] starting ... [2015-01-06 21:29:45,960][INFO][transport] [bi-es1] bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/192.168.83.231:9300]} [2015-01-06 21:29:45,982][INFO][discovery] [bi-es1] bi-cluster1/VyTzSKWnQAS9ZCk971fJEg [2015-01-06 21:29:51,906][WARN][transport.netty] [bi-es1] Message not fully read (request) for [28805963] and action [discovery/zen/join/validate], resetting [2015-01-06 21:29:51,915][INFO][discovery.zen] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ] [2015-01-06 21:29:57,028][WARN][transport.netty] [bi-es1] Message not fully read (request) for [28807127] and action [discovery/zen/join/validate], resetting [2015-01-06 21:29:57,036][INFO][discovery.zen] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ] [2015-01-06 21:30:02,245][WARN][transport.netty] [bi-es1] Message not fully read (request) for [28808254] and action [discovery/zen/join/validate], resetting [2015-01-06 21:30:02,252][INFO][discovery.zen] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ] [2015-01-06 21:30:07,576][WARN][transport.netty] [bi-es1] Message not fully read (request) for [28809377] and action [discovery/zen/join/validate], resetting [2015-01-06 21:30:07,583][INFO][discovery.zen] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ] [2015-01-06 21:30:12,689][WARN][transport.netty] [bi-es1] Message not fully read (request) for [28810050] and action [discovery/zen/join/validate], resetting [2015-01-06 21:30:12,696][INFO][discovery.zen] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ] [2015-01-06 21:30:15,989][WARN][discovery] [bi-es1] waited for 30s and no initial state was set by the discovery [2015-01-06 21:30:16,002][INFO][http] [bi-es1] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/192.168.83.231:9200]} [2015-01-06 21:30:16,003][INFO][node] [bi-es1] started [2015-01-06 21:30:18,277][WARN][transport.netty] [bi-es1] Message not fully read (request) for [28811164] and action [discovery/zen/join/validate], resetting [2015-01-06 21:30:18,292][INFO][discovery.zen] [bi-es1] failed to send join request to master [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; nested: ElasticsearchIllegalArgumentException[No custom index metadata factory registered for type [rivers]]; ] [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no known master node, scheduling a retry I have tried removing all plugins from the upgraded node and that results in same errors. I have tried closing the \_river index and removing all defined rivers (wasn't hopeful that would work but gave it a shot). I am not sure what else to try - thoughts? ideas? I am safely downgraded back to 1.3.2 and node rejoined and things seem good except still on 1.3.2. Would prefer not to do a full cluster shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would solve this? I have had issues in past with upgrades/jdbc river plugin (really love the plugin though) so my brain is focusing on that and might not be the issue? I used same process on a test cluster that I just realized did not have the jdbc plugin installed on it and that went smooth. Any help would be greatly appreciated! Thanks Ben -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 8, 2015, 4:24pm UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/2 "2015-01-08T16:24:55Z")
Hello Ben, Maybe it works if you uninstall the plugin from one node at a time and do a rolling restart (sticking to 1.3.2), then do the upgrade with another rolling restart, then install the plugin back again with yet another rolling restart? I would understand if you said "no way I do 3 restarts!" 🙂 But maybe this will help in future: [GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins](https://github.com/jprante/elasticsearch-plugin-deploy) ## Best regards, Radu Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg [benberg86@gmail.com](mailto:benberg86@gmail.com) wrote: > Hello, > I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I > upgraded the first node,removed and reinstalled latest versions of plugins > > - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version > 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the > errors below in logs on upgraded node and node does not join cluster. If i > downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin > version 1.3.0.4 the node properly joins the cluster. > Errors from logs: > [2015-01-06 21:29:39,970][INFO][node] [bi-es1] > version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z] > [2015-01-06 21:29:39,971][INFO][node] [bi-es1] > initializing ... > [2015-01-06 21:29:40,009][INFO][plugins] [bi-es1] > loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, > paramedic, HQ, whatson] > [2015-01-06 21:29:45,587][INFO][node] [bi-es1] > initialized > [2015-01-06 21:29:45,588][INFO][node] [bi-es1] > starting ... > [2015-01-06 21:29:45,960][INFO][transport] [bi-es1] > bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/ > 192.168.83.231:9300]} > [2015-01-06 21:29:45,982][INFO][discovery] [bi-es1] > bi-cluster1/VyTzSKWnQAS9ZCk971fJEg > [2015-01-06 21:29:51,906][WARN][transport.netty] [bi-es1] > Message not fully read (request) for [28805963] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:29:51,915][INFO][discovery.zen] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:29:57,028][WARN][transport.netty] [bi-es1] > Message not fully read (request) for [28807127] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:29:57,036][INFO][discovery.zen] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:02,245][WARN][transport.netty] [bi-es1] > Message not fully read (request) for [28808254] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:30:02,252][INFO][discovery.zen] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:07,576][WARN][transport.netty] [bi-es1] > Message not fully read (request) for [28809377] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:30:07,583][INFO][discovery.zen] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:12,689][WARN][transport.netty] [bi-es1] > Message not fully read (request) for [28810050] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:30:12,696][INFO][discovery.zen] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:15,989][WARN][discovery] [bi-es1] > waited for 30s and no initial state was set by the discovery > [2015-01-06 21:30:16,002][INFO][http] [bi-es1] > bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/ > 192.168.83.231:9200]} > [2015-01-06 21:30:16,003][INFO][node] [bi-es1] > started > [2015-01-06 21:30:18,277][WARN][transport.netty] [bi-es1] > Message not fully read (request) for [28811164] and action > [discovery/zen/join/validate], resetting > [2015-01-06 21:30:18,292][INFO][discovery.zen] [bi-es1] > failed to send join request to master > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > nested: ElasticsearchIllegalArgumentException[No custom index metadata > factory registered for type [rivers]]; ] > [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no > known master node, scheduling a retry > > I have tried removing all plugins from the upgraded node and that results > in same errors. I have tried closing the \_river index and removing all > defined rivers (wasn't hopeful that would work but gave it a shot). > > I am not sure what else to try - thoughts? ideas? > > I am safely downgraded back to 1.3.2 and node rejoined and things seem > good except still on 1.3.2. Would prefer not to do a full cluster shutdown > to upgrade all nodes and jdbc plugin but that seems maybe it would solve > this? > > I have had issues in past with upgrades/jdbc river plugin (really love the > plugin though) so my brain is focusing on that and might not be the issue? > I used same process on a test cluster that I just realized did not have the > jdbc plugin installed on it and that went smooth. > > Any help would be greatly appreciated! > Thanks > Ben > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ben\_Berg](https://avatars.discourse-cdn.com/v4/letter/b/8baadc/32.png) [@Ben\_Berg](https://discuss.elastic.co/u/Ben_Berg)\ **Post date:** [January 8, 2015, 5:01pm UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/3 "2015-01-08T17:01:58Z")
Thanks for the reply! That is a good idea to try, but another problem with this is that when i uninstall the plugins and keep at version 1.3.2 the node will not join cluster either. We upgraded to java 8 right before starting this upgrade and thinking it may have something to do with that - also noticed that somehow JAVA\_HOME was not defined - working on setting that up across the cluster and will attempt to remove plugin while on 1.3.2 and get node to join cluster - if that does not work I will downgrade to java 7 and try again. Thanks again for reply, insights and huge thanks for link to the plugin repo - looks awesome and will probably try it after sorting this out. Will update with findings later today hopefully. Ben -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b5dfdbb1-507e-4032-9ec2-40d91266432b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b5dfdbb1-507e-4032-9ec2-40d91266432b%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ben\_Berg](https://avatars.discourse-cdn.com/v4/letter/b/8baadc/32.png) [@Ben\_Berg](https://discuss.elastic.co/u/Ben_Berg)\ **Post date:** [January 8, 2015, 8:28pm UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/4 "2015-01-08T20:28:52Z")
Ended up finding out that the jdbc plugin requires a full cluster shutdown for restart - [https://github.com/jprante/elasticsearch-river-jdbc/issues/433](https://github.com/jprante/elasticsearch-river-jdbc/issues/433) Going to be doing that in maintenance window tomorrow and will verify that worked. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/60577717-944f-4b13-b66d-124cfb2e7b39%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/60577717-944f-4b13-b66d-124cfb2e7b39%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 8, 2015, 10:28pm UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/5 "2015-01-08T22:28:51Z")
Hi Radu, the deploy plugin is somewhat limited to "restartable" plugins as it can only restart services of a plugin, i.e. lifecycle components with carefully implemented doStart()/doStop() methods. Most plugins come with modules that add REST endpoints, actions, parsers, functions etc. as modules by the onModule...() mechanism, so they get "baked" into the ES node as immutable objects before the node is going up. This means, the deploy plugin can not change or undo this behavior. (Maybe some evil hackery will do to a certain degree) For my use case, I embed a Ratpack server and start my own HTTP web server that does not use baked-in ES modules, just a restartable service. It is really delightful to just use a single curl PUT command for rapid prototyping a Ratpack web app embedded in ES without node restart. A restartable plugin is: > **[GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for...](https://github.com/jprante/elasticsearch-plugin-ratpack)** > > Elasticsearch plugin for embedding Ratpack http://ratpack.io - GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for embedding Ratpack http://ratpack.io Jörg On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) wrote: > Hello Ben, > > Maybe it works if you uninstall the plugin from one node at a time and do > a rolling restart (sticking to 1.3.2), then do the upgrade with another > rolling restart, then install the plugin back again with yet another > rolling restart? > > I would understand if you said "no way I do 3 restarts!" 🙂 But maybe this > will help in future: > [GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins](https://github.com/jprante/elasticsearch-plugin-deploy) > > ## Best regards, Radu > > Performance Monitoring \* Log Analytics \* Search Analytics > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg [benberg86@gmail.com](mailto:benberg86@gmail.com) wrote: > > > Hello, > > I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I > > upgraded the first node,removed and reinstalled latest versions of plugins > > > > - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version > > 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the > > errors below in logs on upgraded node and node does not join cluster. If i > > downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin > > version 1.3.0.4 the node properly joins the cluster. > > Errors from logs: > > [2015-01-06 21:29:39,970][INFO][node] [bi-es1] > > version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z] > > [2015-01-06 21:29:39,971][INFO][node] [bi-es1] > > initializing ... > > [2015-01-06 21:29:40,009][INFO][plugins] [bi-es1] > > loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, > > paramedic, HQ, whatson] > > [2015-01-06 21:29:45,587][INFO][node] [bi-es1] > > initialized > > [2015-01-06 21:29:45,588][INFO][node] [bi-es1] > > starting ... > > [2015-01-06 21:29:45,960][INFO][transport] [bi-es1] > > bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/ > > 192.168.83.231:9300]} > > [2015-01-06 21:29:45,982][INFO][discovery] [bi-es1] > > bi-cluster1/VyTzSKWnQAS9ZCk971fJEg > > [2015-01-06 21:29:51,906][WARN][transport.netty] [bi-es1] > > Message not fully read (request) for [28805963] and action > > [discovery/zen/join/validate], resetting > > [2015-01-06 21:29:51,915][INFO][discovery.zen] [bi-es1] > > failed to send join request to master > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > factory registered for type [rivers]]; ] > > [2015-01-06 21:29:57,028][WARN][transport.netty] [bi-es1] > > Message not fully read (request) for [28807127] and action > > [discovery/zen/join/validate], resetting > > [2015-01-06 21:29:57,036][INFO][discovery.zen] [bi-es1] > > failed to send join request to master > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > factory registered for type [rivers]]; ] > > [2015-01-06 21:30:02,245][WARN][transport.netty] [bi-es1] > > Message not fully read (request) for [28808254] and action > > [discovery/zen/join/validate], resetting > > [2015-01-06 21:30:02,252][INFO][discovery.zen] [bi-es1] > > failed to send join request to master > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > factory registered for type [rivers]]; ] > > [2015-01-06 21:30:07,576][WARN][transport.netty] [bi-es1] > > Message not fully read (request) for [28809377] and action > > [discovery/zen/join/validate], resetting > > [2015-01-06 21:30:07,583][INFO][discovery.zen] [bi-es1] > > failed to send join request to master > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > factory registered for type [rivers]]; ] > > [2015-01-06 21:30:12,689][WARN][transport.netty] [bi-es1] > > Message not fully read (request) for [28810050] and action > > [discovery/zen/join/validate], resetting > > [2015-01-06 21:30:12,696][INFO][discovery.zen] [bi-es1] > > failed to send join request to master > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > factory registered for type [rivers]]; ] > > [2015-01-06 21:30:15,989][WARN][discovery] [bi-es1] > > waited for 30s and no initial state was set by the discovery > > [2015-01-06 21:30:16,002][INFO][http] [bi-es1] > > bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/ > > 192.168.83.231:9200]} > > [2015-01-06 21:30:16,003][INFO][node] [bi-es1] > > started > > [2015-01-06 21:30:18,277][WARN][transport.netty] [bi-es1] > > Message not fully read (request) for [28811164] and action > > [discovery/zen/join/validate], resetting > > [2015-01-06 21:30:18,292][INFO][discovery.zen] [bi-es1] > > failed to send join request to master > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > factory registered for type [rivers]]; ] > > [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no > > known master node, scheduling a retry > > > > I have tried removing all plugins from the upgraded node and that results > > in same errors. I have tried closing the \_river index and removing all > > defined rivers (wasn't hopeful that would work but gave it a shot). > > > > I am not sure what else to try - thoughts? ideas? > > > > I am safely downgraded back to 1.3.2 and node rejoined and things seem > > good except still on 1.3.2. Would prefer not to do a full cluster shutdown > > to upgrade all nodes and jdbc plugin but that seems maybe it would solve > > this? > > > > I have had issues in past with upgrades/jdbc river plugin (really love > > the plugin though) so my brain is focusing on that and might not be the > > issue? I used same process on a test cluster that I just realized did not > > have the jdbc plugin installed on it and that went smooth. > > > > Any help would be greatly appreciated! > > Thanks > > Ben > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 9, 2015, 8:01am UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/6 "2015-01-09T08:01:34Z")
Thanks Ben and Joerg. Too bad there was no good solution this particular upgrade, at least we all learned something (well, aside from Joerg who knew it all from the start :D). Best regards, Radu -- Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Fri, Jan 9, 2015 at 12:28 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > Hi Radu, > > the deploy plugin is somewhat limited to "restartable" plugins as it can > only restart services of a plugin, i.e. lifecycle components with carefully > implemented doStart()/doStop() methods. > > Most plugins come with modules that add REST endpoints, actions, parsers, > functions etc. as modules by the onModule...() mechanism, so they get > "baked" into the ES node as immutable objects before the node is going up. > This means, the deploy plugin can not change or undo this behavior. (Maybe > some evil hackery will do to a certain degree) > > For my use case, I embed a Ratpack server and start my own HTTP web server > that does not use baked-in ES modules, just a restartable service. It is > really delightful to just use a single curl PUT command for rapid > prototyping a Ratpack web app embedded in ES without node restart. > > A restartable plugin is: > > [GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for embedding Ratpack http://ratpack.io](https://github.com/jprante/elasticsearch-plugin-ratpack) > > Jörg > > On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) > wrote: > > > Hello Ben, > > > > Maybe it works if you uninstall the plugin from one node at a time and do > > a rolling restart (sticking to 1.3.2), then do the upgrade with another > > rolling restart, then install the plugin back again with yet another > > rolling restart? > > > > I would understand if you said "no way I do 3 restarts!" 🙂 But maybe > > this will help in future: > > [GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins](https://github.com/jprante/elasticsearch-plugin-deploy) > > > > ## Best regards, Radu > > > > Performance Monitoring \* Log Analytics \* Search Analytics > > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > > > On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg [benberg86@gmail.com](mailto:benberg86@gmail.com) wrote: > > > > > Hello, > > > I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I > > > upgraded the first node,removed and reinstalled latest versions of plugins > > > > > > - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version > > > 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the > > > errors below in logs on upgraded node and node does not join cluster. If i > > > downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin > > > version 1.3.0.4 the node properly joins the cluster. > > > Errors from logs: > > > [2015-01-06 21:29:39,970][INFO][node] [bi-es1] > > > version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z] > > > [2015-01-06 21:29:39,971][INFO][node] [bi-es1] > > > initializing ... > > > [2015-01-06 21:29:40,009][INFO][plugins] [bi-es1] > > > loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, > > > paramedic, HQ, whatson] > > > [2015-01-06 21:29:45,587][INFO][node] [bi-es1] > > > initialized > > > [2015-01-06 21:29:45,588][INFO][node] [bi-es1] > > > starting ... > > > [2015-01-06 21:29:45,960][INFO][transport] [bi-es1] > > > bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/ > > > 192.168.83.231:9300]} > > > [2015-01-06 21:29:45,982][INFO][discovery] [bi-es1] > > > bi-cluster1/VyTzSKWnQAS9ZCk971fJEg > > > [2015-01-06 21:29:51,906][WARN][transport.netty] [bi-es1] > > > Message not fully read (request) for [28805963] and action > > > [discovery/zen/join/validate], resetting > > > [2015-01-06 21:29:51,915][INFO][discovery.zen] [bi-es1] > > > failed to send join request to master > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > factory registered for type [rivers]]; ] > > > [2015-01-06 21:29:57,028][WARN][transport.netty] [bi-es1] > > > Message not fully read (request) for [28807127] and action > > > [discovery/zen/join/validate], resetting > > > [2015-01-06 21:29:57,036][INFO][discovery.zen] [bi-es1] > > > failed to send join request to master > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > factory registered for type [rivers]]; ] > > > [2015-01-06 21:30:02,245][WARN][transport.netty] [bi-es1] > > > Message not fully read (request) for [28808254] and action > > > [discovery/zen/join/validate], resetting > > > [2015-01-06 21:30:02,252][INFO][discovery.zen] [bi-es1] > > > failed to send join request to master > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > factory registered for type [rivers]]; ] > > > [2015-01-06 21:30:07,576][WARN][transport.netty] [bi-es1] > > > Message not fully read (request) for [28809377] and action > > > [discovery/zen/join/validate], resetting > > > [2015-01-06 21:30:07,583][INFO][discovery.zen] [bi-es1] > > > failed to send join request to master > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > factory registered for type [rivers]]; ] > > > [2015-01-06 21:30:12,689][WARN][transport.netty] [bi-es1] > > > Message not fully read (request) for [28810050] and action > > > [discovery/zen/join/validate], resetting > > > [2015-01-06 21:30:12,696][INFO][discovery.zen] [bi-es1] > > > failed to send join request to master > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > factory registered for type [rivers]]; ] > > > [2015-01-06 21:30:15,989][WARN][discovery] [bi-es1] > > > waited for 30s and no initial state was set by the discovery > > > [2015-01-06 21:30:16,002][INFO][http] [bi-es1] > > > bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/ > > > 192.168.83.231:9200]} > > > [2015-01-06 21:30:16,003][INFO][node] [bi-es1] > > > started > > > [2015-01-06 21:30:18,277][WARN][transport.netty] [bi-es1] > > > Message not fully read (request) for [28811164] and action > > > [discovery/zen/join/validate], resetting > > > [2015-01-06 21:30:18,292][INFO][discovery.zen] [bi-es1] > > > failed to send join request to master > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > factory registered for type [rivers]]; ] > > > [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] no > > > known master node, scheduling a retry > > > > > > I have tried removing all plugins from the upgraded node and that > > > results in same errors. I have tried closing the \_river index and removing > > > all defined rivers (wasn't hopeful that would work but gave it a shot). > > > > > > I am not sure what else to try - thoughts? ideas? > > > > > > I am safely downgraded back to 1.3.2 and node rejoined and things seem > > > good except still on 1.3.2. Would prefer not to do a full cluster shutdown > > > to upgrade all nodes and jdbc plugin but that seems maybe it would solve > > > this? > > > > > > I have had issues in past with upgrades/jdbc river plugin (really love > > > the plugin though) so my brain is focusing on that and might not be the > > > issue? I used same process on a test cluster that I just realized did not > > > have the jdbc plugin installed on it and that went smooth. > > > > > > Any help would be greatly appreciated! > > > Thanks > > > Ben > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com) > > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > . > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX\_7DAqGROuo0A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 9, 2015, 8:20am UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/7 "2015-01-09T08:20:20Z")
No I didn't ☹ I still struggle with stateful plugins - there is no good solution I know of and that is the reason of trouble. Jörg On Fri, Jan 9, 2015 at 9:01 AM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) wrote: > Thanks Ben and Joerg. Too bad there was no good solution this particular > upgrade, at least we all learned something (well, aside from Joerg who knew > it all from the start :D). > > Best regards, > Radu > > -- > Performance Monitoring \* Log Analytics \* Search Analytics > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > On Fri, Jan 9, 2015 at 12:28 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > Hi Radu, > > > > the deploy plugin is somewhat limited to "restartable" plugins as it can > > only restart services of a plugin, i.e. lifecycle components with carefully > > implemented doStart()/doStop() methods. > > > > Most plugins come with modules that add REST endpoints, actions, parsers, > > functions etc. as modules by the onModule...() mechanism, so they get > > "baked" into the ES node as immutable objects before the node is going up. > > This means, the deploy plugin can not change or undo this behavior. (Maybe > > some evil hackery will do to a certain degree) > > > > For my use case, I embed a Ratpack server and start my own HTTP web > > server that does not use baked-in ES modules, just a restartable service. > > It is really delightful to just use a single curl PUT command for rapid > > prototyping a Ratpack web app embedded in ES without node restart. > > > > A restartable plugin is: > > > > [GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for embedding Ratpack http://ratpack.io](https://github.com/jprante/elasticsearch-plugin-ratpack) > > > > Jörg > > > > On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe \<[radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) > > > > > wrote: > > > > > Hello Ben, > > > > > > Maybe it works if you uninstall the plugin from one node at a time and > > > do a rolling restart (sticking to 1.3.2), then do the upgrade with another > > > rolling restart, then install the plugin back again with yet another > > > rolling restart? > > > > > > I would understand if you said "no way I do 3 restarts!" 🙂 But maybe > > > this will help in future: > > > [GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins](https://github.com/jprante/elasticsearch-plugin-deploy) > > > > > > ## Best regards, Radu > > > > > > Performance Monitoring \* Log Analytics \* Search Analytics > > > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > > > > > On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg [benberg86@gmail.com](mailto:benberg86@gmail.com) wrote: > > > > > > > Hello, > > > > I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I > > > > upgraded the first node,removed and reinstalled latest versions of plugins > > > > > > > > - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version > > > > 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the > > > > errors below in logs on upgraded node and node does not join cluster. If i > > > > downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin > > > > version 1.3.0.4 the node properly joins the cluster. > > > > Errors from logs: > > > > [2015-01-06 21:29:39,970][INFO][node] [bi-es1] > > > > version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z] > > > > [2015-01-06 21:29:39,971][INFO][node] [bi-es1] > > > > initializing ... > > > > [2015-01-06 21:29:40,009][INFO][plugins] [bi-es1] > > > > loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, > > > > paramedic, HQ, whatson] > > > > [2015-01-06 21:29:45,587][INFO][node] [bi-es1] > > > > initialized > > > > [2015-01-06 21:29:45,588][INFO][node] [bi-es1] > > > > starting ... > > > > [2015-01-06 21:29:45,960][INFO][transport] [bi-es1] > > > > bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/ > > > > 192.168.83.231:9300]} > > > > [2015-01-06 21:29:45,982][INFO][discovery] [bi-es1] > > > > bi-cluster1/VyTzSKWnQAS9ZCk971fJEg > > > > [2015-01-06 21:29:51,906][WARN][transport.netty] [bi-es1] > > > > Message not fully read (request) for [28805963] and action > > > > [discovery/zen/join/validate], resetting > > > > [2015-01-06 21:29:51,915][INFO][discovery.zen] [bi-es1] > > > > failed to send join request to master > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > factory registered for type [rivers]]; ] > > > > [2015-01-06 21:29:57,028][WARN][transport.netty] [bi-es1] > > > > Message not fully read (request) for [28807127] and action > > > > [discovery/zen/join/validate], resetting > > > > [2015-01-06 21:29:57,036][INFO][discovery.zen] [bi-es1] > > > > failed to send join request to master > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > factory registered for type [rivers]]; ] > > > > [2015-01-06 21:30:02,245][WARN][transport.netty] [bi-es1] > > > > Message not fully read (request) for [28808254] and action > > > > [discovery/zen/join/validate], resetting > > > > [2015-01-06 21:30:02,252][INFO][discovery.zen] [bi-es1] > > > > failed to send join request to master > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > factory registered for type [rivers]]; ] > > > > [2015-01-06 21:30:07,576][WARN][transport.netty] [bi-es1] > > > > Message not fully read (request) for [28809377] and action > > > > [discovery/zen/join/validate], resetting > > > > [2015-01-06 21:30:07,583][INFO][discovery.zen] [bi-es1] > > > > failed to send join request to master > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > factory registered for type [rivers]]; ] > > > > [2015-01-06 21:30:12,689][WARN][transport.netty] [bi-es1] > > > > Message not fully read (request) for [28810050] and action > > > > [discovery/zen/join/validate], resetting > > > > [2015-01-06 21:30:12,696][INFO][discovery.zen] [bi-es1] > > > > failed to send join request to master > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > factory registered for type [rivers]]; ] > > > > [2015-01-06 21:30:15,989][WARN][discovery] [bi-es1] > > > > waited for 30s and no initial state was set by the discovery > > > > [2015-01-06 21:30:16,002][INFO][http] [bi-es1] > > > > bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/ > > > > 192.168.83.231:9200]} > > > > [2015-01-06 21:30:16,003][INFO][node] [bi-es1] > > > > started > > > > [2015-01-06 21:30:18,277][WARN][transport.netty] [bi-es1] > > > > Message not fully read (request) for [28811164] and action > > > > [discovery/zen/join/validate], resetting > > > > [2015-01-06 21:30:18,292][INFO][discovery.zen] [bi-es1] > > > > failed to send join request to master > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > factory registered for type [rivers]]; ] > > > > [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] > > > > no known master node, scheduling a retry > > > > > > > > I have tried removing all plugins from the upgraded node and that > > > > results in same errors. I have tried closing the \_river index and removing > > > > all defined rivers (wasn't hopeful that would work but gave it a shot). > > > > > > > > I am not sure what else to try - thoughts? ideas? > > > > > > > > I am safely downgraded back to 1.3.2 and node rejoined and things seem > > > > good except still on 1.3.2. Would prefer not to do a full cluster shutdown > > > > to upgrade all nodes and jdbc plugin but that seems maybe it would solve > > > > this? > > > > > > > > I have had issues in past with upgrades/jdbc river plugin (really love > > > > the plugin though) so my brain is focusing on that and might not be the > > > > issue? I used same process on a test cluster that I just realized did not > > > > have the jdbc plugin installed on it and that went smooth. > > > > > > > > Any help would be greatly appreciated! > > > > Thanks > > > > Ben > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX\_7DAqGROuo0A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX\_7DAqGROuo0A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a\_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [January 9, 2015, 8:22am UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/8 "2015-01-09T08:22:38Z")
Yeah, my general approach so far was to stay away from plugins as much as possible and use external services instead. Something that your JDBC river can do as well, as far as I understand. -- Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) On Fri, Jan 9, 2015 at 10:20 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > No I didn't ☹ > > I still struggle with stateful plugins - there is no good solution I know > of and that is the reason of trouble. > > Jörg > > On Fri, Jan 9, 2015 at 9:01 AM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) > wrote: > > > Thanks Ben and Joerg. Too bad there was no good solution this particular > > upgrade, at least we all learned something (well, aside from Joerg who knew > > it all from the start :D). > > > > Best regards, > > Radu > > > > -- > > Performance Monitoring \* Log Analytics \* Search Analytics > > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > > > On Fri, Jan 9, 2015 at 12:28 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > Hi Radu, > > > > > > the deploy plugin is somewhat limited to "restartable" plugins as it can > > > only restart services of a plugin, i.e. lifecycle components with carefully > > > implemented doStart()/doStop() methods. > > > > > > Most plugins come with modules that add REST endpoints, actions, > > > parsers, functions etc. as modules by the onModule...() mechanism, so they > > > get "baked" into the ES node as immutable objects before the node is going > > > up. This means, the deploy plugin can not change or undo this behavior. > > > (Maybe some evil hackery will do to a certain degree) > > > > > > For my use case, I embed a Ratpack server and start my own HTTP web > > > server that does not use baked-in ES modules, just a restartable service. > > > It is really delightful to just use a single curl PUT command for rapid > > > prototyping a Ratpack web app embedded in ES without node restart. > > > > > > A restartable plugin is: > > > > > > [GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for embedding Ratpack http://ratpack.io](https://github.com/jprante/elasticsearch-plugin-ratpack) > > > > > > Jörg > > > > > > On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe \< > > > [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com)\> wrote: > > > > > > > Hello Ben, > > > > > > > > Maybe it works if you uninstall the plugin from one node at a time and > > > > do a rolling restart (sticking to 1.3.2), then do the upgrade with another > > > > rolling restart, then install the plugin back again with yet another > > > > rolling restart? > > > > > > > > I would understand if you said "no way I do 3 restarts!" 🙂 But maybe > > > > this will help in future: > > > > [GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins](https://github.com/jprante/elasticsearch-plugin-deploy) > > > > > > > > ## Best regards, Radu > > > > > > > > Performance Monitoring \* Log Analytics \* Search Analytics > > > > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > > > > > > > On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg [benberg86@gmail.com](mailto:benberg86@gmail.com) wrote: > > > > > > > > > Hello, > > > > > I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I > > > > > upgraded the first node,removed and reinstalled latest versions of plugins > > > > > > > > > > - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version > > > > > 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the > > > > > errors below in logs on upgraded node and node does not join cluster. If i > > > > > downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin > > > > > version 1.3.0.4 the node properly joins the cluster. > > > > > Errors from logs: > > > > > [2015-01-06 21:29:39,970][INFO][node] [bi-es1] > > > > > version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z] > > > > > [2015-01-06 21:29:39,971][INFO][node] [bi-es1] > > > > > initializing ... > > > > > [2015-01-06 21:29:40,009][INFO][plugins] [bi-es1] > > > > > loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, > > > > > paramedic, HQ, whatson] > > > > > [2015-01-06 21:29:45,587][INFO][node] [bi-es1] > > > > > initialized > > > > > [2015-01-06 21:29:45,588][INFO][node] [bi-es1] > > > > > starting ... > > > > > [2015-01-06 21:29:45,960][INFO][transport] [bi-es1] > > > > > bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/ > > > > > 192.168.83.231:9300]} > > > > > [2015-01-06 21:29:45,982][INFO][discovery] [bi-es1] > > > > > bi-cluster1/VyTzSKWnQAS9ZCk971fJEg > > > > > [2015-01-06 21:29:51,906][WARN][transport.netty] [bi-es1] > > > > > Message not fully read (request) for [28805963] and action > > > > > [discovery/zen/join/validate], resetting > > > > > [2015-01-06 21:29:51,915][INFO][discovery.zen] [bi-es1] > > > > > failed to send join request to master > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > factory registered for type [rivers]]; ] > > > > > [2015-01-06 21:29:57,028][WARN][transport.netty] [bi-es1] > > > > > Message not fully read (request) for [28807127] and action > > > > > [discovery/zen/join/validate], resetting > > > > > [2015-01-06 21:29:57,036][INFO][discovery.zen] [bi-es1] > > > > > failed to send join request to master > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > factory registered for type [rivers]]; ] > > > > > [2015-01-06 21:30:02,245][WARN][transport.netty] [bi-es1] > > > > > Message not fully read (request) for [28808254] and action > > > > > [discovery/zen/join/validate], resetting > > > > > [2015-01-06 21:30:02,252][INFO][discovery.zen] [bi-es1] > > > > > failed to send join request to master > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > factory registered for type [rivers]]; ] > > > > > [2015-01-06 21:30:07,576][WARN][transport.netty] [bi-es1] > > > > > Message not fully read (request) for [28809377] and action > > > > > [discovery/zen/join/validate], resetting > > > > > [2015-01-06 21:30:07,583][INFO][discovery.zen] [bi-es1] > > > > > failed to send join request to master > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > factory registered for type [rivers]]; ] > > > > > [2015-01-06 21:30:12,689][WARN][transport.netty] [bi-es1] > > > > > Message not fully read (request) for [28810050] and action > > > > > [discovery/zen/join/validate], resetting > > > > > [2015-01-06 21:30:12,696][INFO][discovery.zen] [bi-es1] > > > > > failed to send join request to master > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > factory registered for type [rivers]]; ] > > > > > [2015-01-06 21:30:15,989][WARN][discovery] [bi-es1] > > > > > waited for 30s and no initial state was set by the discovery > > > > > [2015-01-06 21:30:16,002][INFO][http] [bi-es1] > > > > > bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/ > > > > > 192.168.83.231:9200]} > > > > > [2015-01-06 21:30:16,003][INFO][node] [bi-es1] > > > > > started > > > > > [2015-01-06 21:30:18,277][WARN][transport.netty] [bi-es1] > > > > > Message not fully read (request) for [28811164] and action > > > > > [discovery/zen/join/validate], resetting > > > > > [2015-01-06 21:30:18,292][INFO][discovery.zen] [bi-es1] > > > > > failed to send join request to master > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > factory registered for type [rivers]]; ] > > > > > [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] > > > > > no known master node, scheduling a retry > > > > > > > > > > I have tried removing all plugins from the upgraded node and that > > > > > results in same errors. I have tried closing the \_river index and removing > > > > > all defined rivers (wasn't hopeful that would work but gave it a shot). > > > > > > > > > > I am not sure what else to try - thoughts? ideas? > > > > > > > > > > I am safely downgraded back to 1.3.2 and node rejoined and things seem > > > > > good except still on 1.3.2. Would prefer not to do a full cluster shutdown > > > > > to upgrade all nodes and jdbc plugin but that seems maybe it would solve > > > > > this? > > > > > > > > > > I have had issues in past with upgrades/jdbc river plugin (really love > > > > > the plugin though) so my brain is focusing on that and might not be the > > > > > issue? I used same process on a test cluster that I just realized did not > > > > > have the jdbc plugin installed on it and that went smooth. > > > > > > > > > > Any help would be greatly appreciated! > > > > > Thanks > > > > > Ben > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX\_7DAqGROuo0A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX\_7DAqGROuo0A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a\_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a\_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [January 9, 2015, 9:28am UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/9 "2015-01-09T09:28:56Z")
I could switch away from plugin mode, but that would mean people must adapt and sell another JDBC bridge component to their boss and worse, that will keep the SPOF weakness of rivers but with more burden of responsibility on myself. Not in my sparetime. Maybe a compromise will work better: external service, but also attached to the ES nodes as a "lightweight" restartable plugin. It looks attractive to me to move over to Groovy and run SQL / JDBC from there. The picture is an enhanced deploy plugin and a general harvester job distribution module with UI, where JDBC is just one option. With the deploy plugin it is easy to move JDBC driver jars and other dependencies around in the cluster, for stop/start, keep state etc. Jörg On Fri, Jan 9, 2015 at 9:22 AM, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) wrote: > Yeah, my general approach so far was to stay away from plugins as much as > possible and use external services instead. Something that your JDBC river > can do as well, as far as I understand. > > -- > Performance Monitoring \* Log Analytics \* Search Analytics > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > On Fri, Jan 9, 2015 at 10:20 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > No I didn't ☹ > > > > I still struggle with stateful plugins - there is no good solution I know > > of and that is the reason of trouble. > > > > Jörg > > > > On Fri, Jan 9, 2015 at 9:01 AM, Radu Gheorghe \<[radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) > > > > > wrote: > > > > > Thanks Ben and Joerg. Too bad there was no good solution this particular > > > upgrade, at least we all learned something (well, aside from Joerg who knew > > > it all from the start :D). > > > > > > Best regards, > > > Radu > > > > > > -- > > > Performance Monitoring \* Log Analytics \* Search Analytics > > > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > > > > > On Fri, Jan 9, 2015 at 12:28 AM, [joergprante@gmail.com](mailto:joergprante@gmail.com) \< > > > [joergprante@gmail.com](mailto:joergprante@gmail.com)\> wrote: > > > > > > > Hi Radu, > > > > > > > > the deploy plugin is somewhat limited to "restartable" plugins as it > > > > can only restart services of a plugin, i.e. lifecycle components with > > > > carefully implemented doStart()/doStop() methods. > > > > > > > > Most plugins come with modules that add REST endpoints, actions, > > > > parsers, functions etc. as modules by the onModule...() mechanism, so they > > > > get "baked" into the ES node as immutable objects before the node is going > > > > up. This means, the deploy plugin can not change or undo this behavior. > > > > (Maybe some evil hackery will do to a certain degree) > > > > > > > > For my use case, I embed a Ratpack server and start my own HTTP web > > > > server that does not use baked-in ES modules, just a restartable service. > > > > It is really delightful to just use a single curl PUT command for rapid > > > > prototyping a Ratpack web app embedded in ES without node restart. > > > > > > > > A restartable plugin is: > > > > > > > > [GitHub - jprante/elasticsearch-plugin-ratpack: Elasticsearch plugin for embedding Ratpack http://ratpack.io](https://github.com/jprante/elasticsearch-plugin-ratpack) > > > > > > > > Jörg > > > > > > > > On Thu, Jan 8, 2015 at 5:24 PM, Radu Gheorghe \< > > > > [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com)\> wrote: > > > > > > > > > Hello Ben, > > > > > > > > > > Maybe it works if you uninstall the plugin from one node at a time and > > > > > do a rolling restart (sticking to 1.3.2), then do the upgrade with another > > > > > rolling restart, then install the plugin back again with yet another > > > > > rolling restart? > > > > > > > > > > I would understand if you said "no way I do 3 restarts!" 🙂 But maybe > > > > > this will help in future: > > > > > [GitHub - jprante/elasticsearch-plugin-deploy: Refreshable Elasticsearch plugins](https://github.com/jprante/elasticsearch-plugin-deploy) > > > > > > > > > > ## Best regards, Radu > > > > > > > > > > Performance Monitoring \* Log Analytics \* Search Analytics > > > > > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > > > > > > > > > On Wed, Jan 7, 2015 at 5:15 AM, Ben Berg [benberg86@gmail.com](mailto:benberg86@gmail.com) wrote: > > > > > > > > > > > Hello, > > > > > > I am attempting to upgrade a 10 node cluster from 1.3.2 to 1.4.2 - I > > > > > > upgraded the first node,removed and reinstalled latest versions of plugins > > > > > > > > > > > > - two non-site plugins are river-twitter (version 2.4.1) and jdbc (version > > > > > > 1.4.0.8 and also tried 1.4.0.7) - and when starting the node I see the > > > > > > errors below in logs on upgraded node and node does not join cluster. If i > > > > > > downgrade to 1.3.2, uninstall plugins and reinstall jdbc river plugin > > > > > > version 1.3.0.4 the node properly joins the cluster. > > > > > > Errors from logs: > > > > > > [2015-01-06 21:29:39,970][INFO][node] [bi-es1] > > > > > > version[1.4.2], pid[5910], build[927caff/2014-12-16T14:11:12Z] > > > > > > [2015-01-06 21:29:39,971][INFO][node] [bi-es1] > > > > > > initializing ... > > > > > > [2015-01-06 21:29:40,009][INFO][plugins] [bi-es1] > > > > > > loaded [river-twitter, jdbc-1.4.0.7-a875ced], sites [head, kopf, bigdesk, > > > > > > paramedic, HQ, whatson] > > > > > > [2015-01-06 21:29:45,587][INFO][node] [bi-es1] > > > > > > initialized > > > > > > [2015-01-06 21:29:45,588][INFO][node] [bi-es1] > > > > > > starting ... > > > > > > [2015-01-06 21:29:45,960][INFO][transport] [bi-es1] > > > > > > bound\_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish\_address {inet[/ > > > > > > 192.168.83.231:9300]} > > > > > > [2015-01-06 21:29:45,982][INFO][discovery] [bi-es1] > > > > > > bi-cluster1/VyTzSKWnQAS9ZCk971fJEg > > > > > > [2015-01-06 21:29:51,906][WARN][transport.netty] [bi-es1] > > > > > > Message not fully read (request) for [28805963] and action > > > > > > [discovery/zen/join/validate], resetting > > > > > > [2015-01-06 21:29:51,915][INFO][discovery.zen] [bi-es1] > > > > > > failed to send join request to master > > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > > factory registered for type [rivers]]; ] > > > > > > [2015-01-06 21:29:57,028][WARN][transport.netty] [bi-es1] > > > > > > Message not fully read (request) for [28807127] and action > > > > > > [discovery/zen/join/validate], resetting > > > > > > [2015-01-06 21:29:57,036][INFO][discovery.zen] [bi-es1] > > > > > > failed to send join request to master > > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > > factory registered for type [rivers]]; ] > > > > > > [2015-01-06 21:30:02,245][WARN][transport.netty] [bi-es1] > > > > > > Message not fully read (request) for [28808254] and action > > > > > > [discovery/zen/join/validate], resetting > > > > > > [2015-01-06 21:30:02,252][INFO][discovery.zen] [bi-es1] > > > > > > failed to send join request to master > > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > > factory registered for type [rivers]]; ] > > > > > > [2015-01-06 21:30:07,576][WARN][transport.netty] [bi-es1] > > > > > > Message not fully read (request) for [28809377] and action > > > > > > [discovery/zen/join/validate], resetting > > > > > > [2015-01-06 21:30:07,583][INFO][discovery.zen] [bi-es1] > > > > > > failed to send join request to master > > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > > factory registered for type [rivers]]; ] > > > > > > [2015-01-06 21:30:12,689][WARN][transport.netty] [bi-es1] > > > > > > Message not fully read (request) for [28810050] and action > > > > > > [discovery/zen/join/validate], resetting > > > > > > [2015-01-06 21:30:12,696][INFO][discovery.zen] [bi-es1] > > > > > > failed to send join request to master > > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > > factory registered for type [rivers]]; ] > > > > > > [2015-01-06 21:30:15,989][WARN][discovery] [bi-es1] > > > > > > waited for 30s and no initial state was set by the discovery > > > > > > [2015-01-06 21:30:16,002][INFO][http] [bi-es1] > > > > > > bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/ > > > > > > 192.168.83.231:9200]} > > > > > > [2015-01-06 21:30:16,003][INFO][node] [bi-es1] > > > > > > started > > > > > > [2015-01-06 21:30:18,277][WARN][transport.netty] [bi-es1] > > > > > > Message not fully read (request) for [28811164] and action > > > > > > [discovery/zen/join/validate], resetting > > > > > > [2015-01-06 21:30:18,292][INFO][discovery.zen] [bi-es1] > > > > > > failed to send join request to master > > > > > > [[bi-es4][-cczOZfKTw-duMdUO\_YSAw][bi-es4][inet[/192.168.83.234:9300]]{master=true}], > > > > > > reason [RemoteTransportException[[bi-es4][inet[/192.168.83.234:9300]][discovery/zen/join]]; > > > > > > nested: RemoteTransportException[[bi-es1][inet[/192.168.83.231:9300]][discovery/zen/join/validate]]; > > > > > > nested: ElasticsearchIllegalArgumentException[No custom index metadata > > > > > > factory registered for type [rivers]]; ] > > > > > > [2015-01-06 21:30:19,406][DEBUG][action.admin.cluster.state] [bi-es1] > > > > > > no known master node, scheduling a retry > > > > > > > > > > > > I have tried removing all plugins from the upgraded node and that > > > > > > results in same errors. I have tried closing the \_river index and removing > > > > > > all defined rivers (wasn't hopeful that would work but gave it a shot). > > > > > > > > > > > > I am not sure what else to try - thoughts? ideas? > > > > > > > > > > > > I am safely downgraded back to 1.3.2 and node rejoined and things > > > > > > seem good except still on 1.3.2. Would prefer not to do a full cluster > > > > > > shutdown to upgrade all nodes and jdbc plugin but that seems maybe it would > > > > > > solve this? > > > > > > > > > > > > I have had issues in past with upgrades/jdbc river plugin (really > > > > > > love the plugin though) so my brain is focusing on that and might not be > > > > > > the issue? I used same process on a test cluster that I just realized did > > > > > > not have the jdbc plugin installed on it and that went smooth. > > > > > > > > > > > > Any help would be greatly appreciated! > > > > > > Thanks > > > > > > Ben > > > > > > > > > > > > -- > > > > > > You received this message because you are subscribed to the Google > > > > > > Groups "elasticsearch" group. > > > > > > To unsubscribe from this group and stop receiving emails from it, > > > > > > send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > > To view this discussion on the web visit > > > > > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com) > > > > > > [https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/ef4334fc-b6c9-4802-ab8a-934989901efd%40googlegroups.com?utm_medium=email&utm_source=footer) > > > > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > > > -- > > > > > You received this message because you are subscribed to the Google > > > > > Groups "elasticsearch" group. > > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > > To view this discussion on the web visit > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com) > > > > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_16LKxArbm7K3xLv0ewbnaOAnZZLQx5ZZqLezssfebkfA%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > > . > > > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > > To view this discussion on the web visit > > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com) > > > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoGsq6W1rkfjaofGFYY%2Bp8eOJBZ3hKr%2BZrxUtV9jP0R%3DHg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > > . > > > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX\_7DAqGROuo0A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com) > > > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX\_7DAqGROuo0A%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3xVB-ZKRm2SC4Dwdv0WxNAERCskCVicX_7DAqGROuo0A%40mail.gmail.com?utm_medium=email&utm_source=footer) > > > . > > > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a\_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com) > > [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a\_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEm8bNRTsJ%3DQEJi5a_otLToz4Fbhf%2BFoU-FnUJnF9gmGg%40mail.gmail.com?utm_medium=email&utm_source=footer) > > . > > > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_3PfMbU6oEtOyoT3zgiG5gbVt973jOf6B6q8ReJ8C0yXg%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEs%3D8BFHS59eXES0gSwY2Z6M1K9Ctz3J7Z1ZZ8CJk1K7g%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEs%3D8BFHS59eXES0gSwY2Z6M1K9Ctz3J7Z1ZZ8CJk1K7g%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:39am UTC](https://discuss.elastic.co/t/upgraded-node-unable-to-join-cluster-while-attempting-cluster-upgrade-from-1-3-2-to-1-4-2/21516/10 "2017-07-06T00:39:54Z")