Migration from ES V6.8 to V7.17 with an additional node

Elastic Stack Elasticsearch
1

Hi there,

I have a V6.8 instance with a ~350 GB index and plan to migrate to ES V7.17.

I read that ES can migrate between major versions, so my idea was to setup a new V7.17 node and let him join to the existing V6.8 node.

This seems to work, as I can see data is transferred into path.data, but there is more data space used on the new node?

The master node shows

[search01 elasticsearch]# du -shx
355G

The new node
[search02 elasticsearch]# du -shx
416G

This is one of my uncertainties.

The other uncertainty is that the new V7-17 kibana instance tells me that it is incompatible with the V6.8 node

{ ...,"message":"This version of Kibana (v7.17.12) is incompatible with the following Elasticsearch nodes in your cluster: v6.8.23 @ 10.2.13.41:9200 (10.2.13.41)"}

In the V6 kibana I can see this node info:

 "nodes" : {
    "count" : {
      "total" : 2,
      "data" : 2,
      "coordinating_only" : 0,
      "master" : 2,
      "ingest" : 2
    },
    "versions" : [
      "6.8.23",
      "7.17.12"
    ],

My goal was to add a V7 node to get the V6 data migrated while my search service gets NOT interrupted, while I migrate the old V6 node also to V7.17

What did I wrong or is there a better way to have smooth migration with no downtime?

Tnx in advance,
Franco

2

This is correct, Kibana needs to be on the same major and minor version of Elasticsearch, ideally on the exact same version, so Kibana 7.17.12 will note be compatible with Elasticsearch 6.8, only with 7.17.*.

If I understood correctly you had a single-node cluster and added a new node to this cluster, but the node you added was on version 7.17, right?

This alone can lead to some issues because a two node cluster does not have resilience, also, depending on how your configuration looks like, if you have any issue on the communication between your nodes you may end up with a split brain scenario, two different cluster, or with an offline cluster.

Can you share your elasticsearch.yml for both your nodes?

Also, when you add a node on a cluster Elasticsearch will balance the shards between the nodes, trying to keep a equal number of shards on each node, if you want to move shards from your node on version 6 you would to exclude this node from allocation and after it is empty you would be able to upgrade and restart it.

But there is a big catch, currently your master seems to be the node on version 6.X, if you shutdown this node your cluster will be down and I don't think that the node on version 7.X will be able to become the new master as you do not have resilience on a 2 node cluster.

I'm not sure that you can do this with downtime, but please share the elasticsearch.yml of both nodes to make it clear on how your cluster is configured.

3

Hello Leandro,

tnx for you answer.

My two configs look like that:

Node 1 - with ES 6.8.23
----------------------------------
discovery.zen.minimum_master_nodes: 1
node.master: true
network.host: 0.0.0.0
network.publish_host: _ens224_
node.name: ncsearch01
cluster.name: NC-ElasticSearch-CL-01
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
path.repo: ['/mnt/esdata/ES_snapshots']
....

Node 2 - with ES 7.17.12
-----------------------------------

network.host: 0.0.0.0
network.publish_host: _ens224_
http.port: 9200
transport.port: 9300
node.name: ncsearch02
discovery.seed_hosts: ['ncsearch01']
cluster.initial_master_nodes: ['ncsearch01', 'ncsearch02']
cluster.name: NC-ElasticSearch-CL-01
path.repo: ['/mnt/esdata/ES_snapshots']

For now, I stopped the new node-2 because I'm totally unsure what I'm doing.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.