Elassandra 6.2.3 to Elasticsearch 7.7.1 data migration process

Elastic Stack Elasticsearch
1

I want to migrate the data from Elassandra 6.2.3 cluster to Elasticsearch 7.7.1 cluster.
Let's suppose that Elasticsearch 7.7.1 has the same index schemas, templates and aliases. (Except maybe changes related to breaking changes)

Few questions:

1. Can I use reindex api? Is it official way to do migration from Elassandra to Elasticseatch?

2. I tried to use logstash with Elasticsearch input/output plugin for this purposes, which is also worked. However, here I have subset of questions:
a. Input ES plugin does is not supports mutual tls as far as I understood, as well as hostname verification setting. Is there any plans to fix that ?
b. Is there any way to pass index pattern or list of indices, instead of index name (index property)
c. Is it faster that reindex option?

3. After migration is done, is there any way to verify that migration finished successfully, except count of documents.
a. I tried to check the _source and saw that it is the same on both clusters. Search result should be the same because index schema the same (I suppose). Is there a way somehow verify that everything is migrated properly? Is it enough to compare _source, is there any point of doing that?
b. When I tried to work with logstash - _id of copied doc was unique in Elasticseaech cluster, however original _id in Elassandra is different (business value, not auto generated). Question why it is like this and is there any way to say to logstash input plugin that _id should be copied as is?
c. Maybe that logstash es plugin verifies the data after somehow?

2

There's no official way to migrate from elassandra as it's not a supported product by Elastic (us).
I also don't know how much support you will get from it here, I don't recall seeing (m)any posts about it over the years.

That said, if you can use a remote reindex then start there. If not, and it supports a snapshot, try restoring that to the new cluster and then do an internal reindex.

Also 7.7 is pretty old, consider going to 7.13, as it is the latest release.

3

Don't sure I have got all the answers, there are plenty of questions. Could you please take a look.
How I can verify that data is ok?

Again, I tried both ways, data is migrated, but I have to be sure that it is indeed property migrated.

4

Are the document counts in both indicies the same?

5

Yep!

6

Then it's likely migrated across. I cannot be certain though, so you should point your application to the new cluster and make sure it works as expected.

7

It is not the check that I would like to do in production. That is why I am asking about the ways of data integrity checks...

Can I write some app that will retrieve doc from both clusters and compare? What to compare, _source or all json with all tech field? Is it appropriate way to do such things?

8

If the index was created on Elasticsearch 6.x you should be able to use snapshot/restore to move it to the new cluster. That will force you to keep settings but would ensure you do not lose any data.

9

Elassandra 6.2.3

10

I have no experience with Elassandra but assume it is using Elasticsearch. If that is the case you should be able to use snapshot/restore as long as the Elasticsearch versions are compatible.

11

Versions are not compatible, snapshot is not supported as per doc of Elassandra.

12

Which version are they using?

13

As i mentioned in description Elassandra 6.2.3, ES 7.7.1

14

Which version of Elasticsearch does Elassandra use? 6.2.3?

15

Yes 6.2.3

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.