Can't join node after upgrade

Elastic Stack Elasticsearch
1

I can't join my upgraded node to my cluster again.

[2018-10-09T19:21:39,386][WARN ][r.suppressed             ] path: /se-tcj-al-2018-10-09/log/c309f128-    9c17-4848-9fb0-1c2ae3a835c5, params: {index=se-tcj-al-2018-10-09, id=c309f128-9c17-4848-9fb0-1c2ae3a835c5, type=log}
org.elasticsearch.cluster.block.ClusterBlockException: blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];[SERVICE_UNAVAILABLE/2/no master];
        at org.elasticsearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:166) ~[elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.handleBlockExceptions(TransportBulkAction.java:386) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation.doRun(TransportBulkAction.java:272) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.action.bulk.TransportBulkAction$BulkOperation$2.onTimeout(TransportBulkAction.java:420) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:576) [elasticsearch-6.3.0.jar:6.3.0]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_05]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_05]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_05]
        Suppressed: org.elasticsearch.discovery.MasterNotDiscoveredException
                at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$4.onTimeout(TransportMasterNodeAction.java:223) ~[elasticsearch-6.3.0.jar:6.3.0]
                at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:317) [elasticsearch-6.3.0.jar:6.3.0]
                at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:244) [elasticsearch-6.3.0.jar:6.3.0]
                at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:576) [elasticsearch-6.3.0.jar:6.3.0]
                at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:625) [elasticsearch-6.3.0.jar:6.3.0]
                at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_05]
                at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_05]
                at java.lang.Thread.run(Thread.java:745) [?:1.8.0_05]
[2018-10-09T19:21:39,565][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2018-10-09T19:21:39,759][DEBUG][o.e.a.a.i.t.g.TransportGetIndexTemplatesAction] [myhost] no known master node, scheduling a retry
[2018-10-09T19:21:40,081][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2018-10-09T19:21:40,200][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] no known master node, scheduling a retry
[2018-10-09T19:21:41,841][INFO ][o.e.d.z.ZenDiscovery     ] [myhost] failed to send join request to master [{anotherhost}{nk1EByLST2GRnjggABJI2w}{zYlpKPhiQDuyhoGE0HU-Ng}{xxx.xxx.xxx.xxx}{xxx.xxx.xxx.xxx:9300}], reason [RemoteTransportException[[anotherhost][xxx.xxx.xxx.xxx:9300][internal:discovery/zen/join]]; nested: IllegalStateException[index [.marvel-es-1-2016.06.15/VlY3XavpTNK5JoVeAciDfQ] version not supported: 2.3.2 minimum compatible index version is: 5.0.0]; ]
[2018-10-09T19:21:42,200][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2018-10-09T19:21:43,690][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] timed out while retrying [indices:admin/create] after failure (timeout [1m])
[2018-10-09T19:21:44,077][DEBUG][o.e.a.a.i.c.TransportCreateIndexAction] [myhost] no known master node, scheduling a retry

I try to upgrade from 5.6.1 to 6.3.

Any ideas why?

2

bump

3

bump

4

It looks like your node has an index (.marvel-es-1-2016.06.15) that was created in Elasticsearch 2.3.2
ES 6.x cannot read indices from 2.x, you need to use the reindex API to convert them to a newer format, or delete them from the cluster.

See: https://www.elastic.co/guide/en/elasticsearch/reference/6.3/setup-upgrade.html#reindexfn

5

I deleted that one, therefore I am a bit startled about the message.

6

I tried it again today and got the same message.

image
image.png800×78 3.97 KB

Sorry for the ugly screenshot. Any ideas how to prevent this?

7

Not quite the same message - this is now saying that the .kibana index was created in Elasticsearch 2.3.2.

8

Basically the same thing. Both indices were created with Elasticsearch 2.3.2.

9

I don't really understand what you're asking. There are indices in your data directory that were created with Elasticsearch 2.3.2; Elasticsearch 6.x does not support such indices. You must reindex these into new indices and delete the old indices before you upgrade. You said that you'd deleted the .marvel-es-1-2016.06.15 index and then received a similar message about the .kibana index. You have to do this for all your 2.x indices before upgrading.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.