Snapshot/Restore VS ReIndexing - Missing documents, zero failures

Elastic Stack Elasticsearch
1

Hi everyone.

A few weeks ago I created a post regarding a data migration I have to do in a customer with a Platinum License. The conclusion was that I needed to go from elastic 6 to 7 then to 8.12 in order to have the data fully migrated and accessible.

Right now, in a development environment I'm doing a snapshot & restore from 6 do 7.

Everything seems great. I have an index with 4361960 docs and 738.9mb.

When I try to reindex this index, and create a new one, it then shows 142638 docs and 631.3mb.

If I do the same reindex from remote, same index, I get the same number of records, 142638 and around the same size 633.9mb.

I'm completely lost here... I need to reindex the information in Elastic 7, in order to pass it to Elastic 8, because of the archive feature not being support by the Platinum license.

I have zero failures when doing the reindex process.

From 6 to 7, is there something that is being super restructured, eliminating the need for millions of other docs?

Thanks for your time!

Kind regards.

2

It is 1:1 reindex? Like one source index to one destination index or it is N:1, as using a wildcard in the source index?

Can you run this request on Kibana Dev tools on both your source and destination cluster?

GET _cat/indices/index-name?v

Also, what was the Reindex request you run? Since it not much data, can you run it again and share both the request and the response?

3

(this is a development/test environment, only 1 node for Elastic 6, and another node for elastic 7)

Ok so...

  • 20240202 is the snapshot from 6, restored directly in 7. Same number of docs and same size between both instances (I've checked!).
  • 20240202_8_reindextst is the reindex process but getting the data from 6 into the 7 instance.
  • 20240202_8 is the reindex process being ran locally

indexes
indexes1052×67 60.3 KB

This is what I'm running (sorry but I don't have access to Kibana frontend at the moment)

_reindex
_reindex992×436 9.67 KB

And for the snapshot restore, I'm doing the entire process and then running the restore command, which seems to be working since the number of docs is the same between both instances, 4361960:

curl -u elastic:$ELASTICPASS -X POST "$HOSTNAME:9200/_snapshot/repository/XXXXXXXX_XXX_20240202/_restore" -H 'Content-Type: application/json' -d '{
  "indices": "XXXXXXXX_XXX_20240202",
  "ignore_unavailable": true,
  "include_global_state": false
}'
4

Did you apply the same mapping in the destination cluster?
I'm feeling that you had nested fields but you don't set the mapping accordingly.

5

Hi David!

Thanks for the insight!

To tell you the truth I don't even now what mappings are... I'm gonna learn and investigate about the concept and get back to you as soon as possible.

In my mind, snapshot-restore + reindex would be simple processes and as a sysadmin I didn't have to deal with the data schemas.

But give me a few hours to learn about it, compare what was done in Elastic 6 and I'll get back at you.

Thanks!

6

So snapshot/restore is the "db admin" task you'd look for.
It's IMO an admin task.

Reindex is:

  • Search for documents
  • For each document, read the original JSON from the _source field
  • send that json to the new index

I consider that more as a DEV task than an admin task.

If your index has not been configured (same as in RDBMS, if the DBA did not create the destination table), then Elasticsearch tries to guess and creates a default schema.

You can run on the source cluster:

GET /INDEXNAME/

And use that info to create the destination index. See Create index API | Elasticsearch Guide [8.12] | Elastic

7

The thing is... I need to migrate the data from elastic 6.X to 8.12. And to do that, with Platinum License, I have to pass through Elastic 7.X.

(Only the Enterprise license is capable of dealing with the Archive data, in this case, the jump from 6 to 8.12)

I've already tested the Snapshot-restore process, and it works perfectly from 6 to 7. But when I do the same process from 7 to 8, I still get the "archive" error/warning on 8.12.

That's why I was trying to reindex the information on Elastic 7 to see if it needed some changes on the data (at least that was my idea....)

I've tried reindex a smaller index and I still get missing documents like it goes from the original 11000 to 400 in the new reindexed index.

On my mind... snapshot-restore 6 to 7 and 7 to 8 should have done the trick... but then again... when I reach 8... it still gives me the archive error. :confused:

8

Once you are on 7.17, open the upgrade assistant.
It will help you to update old 6.x indices to the new format.

See Upgrade Elasticsearch | Elasticsearch Guide [8.12] | Elastic

This enables you to use the Upgrade Assistant to identify and resolve issues, reindex indices created before 7.0, and then perform a rolling upgrade.

9

So basically... what you are saying is that, my snapshot/restored index from 6 to 7 that I think it's correct (the one with 4 million docs), that I should be able to do something with it on the Upgrade Assistant menu in Elastic 7/Kibana?

I've already asked the network crew to open up communication from "my workstation" to the Elastic 7 node 5601 port. I'm still waiting.

Got to try that idea...

10

If you have access to Elasticsearch then you can still get the mapping of the indices as asked before.

For exampl run a curl to your elasticsearch node on the endpoint $HOSTNAME:9200/index-name/_mapping

This will show you what is the mapping of the index, run this against your cluster on version 6 for one of the index and them against your cluster on version 7 for the same reindexed index, this will help to compare what are the differences.

11

It's huge... the mapping from top to bottom has more than 4000 lines on 6, and on 7 after reindex process into a new index, it shows 2000 lines.

12

This already indicates a possible issue.

Can you share the mapping? Put them on a gist on github because of the size.

13

Search the mappings for the word nested and see if the count match.

1 Like
14

On the Snapshot(6)-Restore(7) Full Index, the one with 4 million docs, on elastic 7 it shows 26 nested words.

On the reindex index, the one with 142638 records, it shows... ZERO "nested" words.

  • Snapshot(6)-Restore(7) index = 4361960 docs - 26 nested words counted in the mapping
  • Reindex index = 142638 docs - 0 nested words counted in the mapping

indexes
indexes1052×67 60.3 KB

15

It makes sense now as nested fields are counted as individual documents.

Before doing the reindex did you create the template for your index on the destination cluster?

You need to use the same template/mapping.

You can get a list of the templates on the endpoint GET /_template/ and the individual template with GET /_template/template-name.

You will need to find what is the template used by the indices and copy and apply them on the destination index.

1 Like
16

Is it possible that I don't have any templates on Elastic 6? Cause it really feels like there are none regarding normal indexes...
image

Also if a run the following:
GET /xxxxxx_xxx_20240202/_settings

I don't see any "template" tag.

And again, I'm not a developer :frowning:

17

Which path do you want to follow? I'm a bit lost...

  • Using Snapshot/Restore?
  • Using reindex from remote?

Snapshot/Restore

Restore in 7.17, then open the migration assistant and follow the instructions. Then upgrade the 7.17 cluster to latest 8.x

Reindex from remote

In the old cluster, run:

GET /INDEXNAME

Take the output of this and run this in the 8.x cluster:

DELETE /INDEXNAME
PUT /INDEXNAME
{
   // The json you got from the previous step
}

Then call the reindex from remote call

That's basically the 2 options.

Let us know if something is unclear or does not work...
Note that the call GET /INDEXNAME might have some not desired metadata... But if you don't know, share here the output and we'll help.

18

Hi David.

It's not you who is lost. It's me :smiley:

Since we can't jump 2 Elastic versions (from 6 to 8) by doing the Snapshot-Restore method while having the platinum license, I thought that the only way to solve this was to create a new Elastic 7 instance, snapshot-restore from 6 to 7, and then run the reindex method to "change" the data so it could be (then again) snapshot-restored from 7 to 8.

This was my initial idea, when people said to me that one method to solve the "archive" situation was to pass through 7...

Now it seems, I understood the situation completely wrong.

So, the right method is to snapshot from 6 to restore on 7, and then! upgrade the installation to Elastic 8?

19

Yes. That's right.

20

Well... since Friday I've remade my entire lab environment and I was able to, finally, migrate from 6 to 8 following your insights and advises.

What now is bugging me a lot is the actual process of ReIndexing via Kibana Upgrade Assistant.

If I perform the reindexing process on my Dev Tools, I lose a bunch of docs because of the situation around the "nested" thing.

But if I perform the "same" process in the Upgrade Assistant, everything performs correctly.

The huge problem that I have is that this process, in the WebUI of Kibana, is a manual one.

I have more than a 1000 indices that I need to migrate and therefor to reindex via Upgrade Assistant... and right now, since I need to click a few times to change some check boxes, etc, etc, it all works... but this is not doable for 1000 and more indices.

My question is... if a simple reindex source-destination command doesn't work in dev tools (nested situation makes my docs fly away into the fifth dimension)... and in Kibana Upgrade Assistant everything goes perfectly... how the heck am I supposed to do this for 1000 and more indices???? :crazy_face:

EDIT:
Btw, the upgrade assistant asks for the following changes, which need to be checked/confirmed via manual input:

  • Index replacement - basically reindexing and creating the alias...
  • Mapping replacement - doc with _doc
  • Removing index.soft_deletes.enable (deprecated).

And then... magically... it does its job... /sadface... XD