Can't join node after upgrade


(Andreas Wachter) #1

I can't join my upgraded node to my cluster again.

[2018-10-09T19:21:39,386][WARN ][r.suppressed             ] path: /se-tcj-al-2018-10-09/log/c309f128-    9c17-4848-9fb0-1c2ae3a835c5, params: {index=se-tcj-al-2018-10-09, id=c309f128-9c17-4848-9fb0-1c2ae3a835c5, type=log}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:386) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:272) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:420) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:576) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_05]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_05]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_05]
        Suppressed: org.elasticsearch.discovery.MasterNotDiscoveredException
                at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:223) ~[elasticsearch-6.3.0.jar:6.3.0]
                at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.3.0.jar:6.3.0]
                at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.3.0.jar:6.3.0]
                at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:576) [elasticsearch-6.3.0.jar:6.3.0]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_05]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_05]
                at java.lang.Thread.run(Thread.java:745) [?:1.8.0_05]
[2018-10-09T19:21:39,565][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2018-10-09T19:21:39,759][DEBUG][o.e.a.a.i.t.g.TransportGetIndexTemplatesAction] [myhost] no known master node, scheduling a retry
[2018-10-09T19:21:40,081][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2018-10-09T19:21:40,200][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] no known master node, scheduling a retry
[2018-10-09T19:21:41,841][INFO ][o.e.d.z.ZenDiscovery     ] [myhost] failed to send join request to master [{anotherhost}{nk1EByLST2GRnjggABJI2w}{zYlpKPhiQDuyhoGE0HU-Ng}{xxx.xxx.xxx.xxx}{xxx.xxx.xxx.xxx:9300}], reason [RemoteTransportException[[anotherhost][xxx.xxx.xxx.xxx:9300][internal:discovery/zen/join]]; nested: IllegalStateException[index [.marvel-es-1-2016.06.15/VlY3XavpTNK5JoVeAciDfQ] version not supported: 2.3.2 minimum compatible index version is: 5.0.0]; ]
[2018-10-09T19:21:42,200][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2018-10-09T19:21:43,690][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2018-10-09T19:21:44,077][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] no known master node, scheduling a retry

I try to upgrade from 5.6.1 to 6.3.

Any ideas why?


(Andreas Wachter) #2

bump


(Andreas Wachter) #3

bump


(Tim Vernum) #4

It looks like your node has an index (.marvel-es-1-2016.06.15) that was created in Elasticsearch 2.3.2
ES 6.x cannot read indices from 2.x, you need to use the reindex API to convert them to a newer format, or delete them from the cluster.

See: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/setup-upgrade.html#reindexfn


(Andreas Wachter) #5

I deleted that one, therefore I am a bit startled about the message.


(Andreas Wachter) #6

I tried it again today and got the same message.


Sorry for the ugly screenshot. Any ideas how to prevent this?


(David Turner) #7

Not quite the same message - this is now saying that the .kibana index was created in Elasticsearch 2.3.2.


(Andreas Wachter) #8

Basically the same thing. Both indices were created with Elasticsearch 2.3.2.


(David Turner) #9

I don't really understand what you're asking. There are indices in your data directory that were created with Elasticsearch 2.3.2; Elasticsearch 6.x does not support such indices. You must reindex these into new indices and delete the old indices before you upgrade. You said that you'd deleted the .marvel-es-1-2016.06.15 index and then received a similar message about the .kibana index. You have to do this for all your 2.x indices before upgrading.


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.