# Snapshot/Restore VS ReIndexing - Missing documents, zero failures **URL:** \ **Category:** Elasticsearch\ **Tags:** reindex\ **Created:** [February 29, 2024, 12:51pm UTC](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448 "2024-02-29T12:51:19Z")\ **Posts on this page:** 8\ **Page:** 2
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 4, 2024, 3:19pm UTC](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448/21 "2024-03-04T15:19:39Z")
> [@Joao\_Barreto](#): > > My question is... if a simple reindex source-destination command doesn't work in dev tools (nested situation makes my docs fly away into the fifth dimension)... The documents do not vanish. The document count in the cat indices API includes internal nested documents, which are not present in the new index as the mappings are not the same. If you use the [count API](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/search-count.html) against both indices you should see that the indices contains the same number of documents. If you are writing a script you need to do the following for each index: - Fetch the existing mapping from the index you are reindexing from - Create the new index you are to reindex into with the mapping extracted in the previous step - Reindex ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 4, 2024, 3:38pm UTC](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448/22 "2024-03-04T15:38:14Z")
I think that the upgrade assistant actually only run a "local reindex" which is actually an update by query: ```auto POST my-index-000001/_update_by_query?conflicts=proceed ``` Or a force merge. I don't remember exactly. ```auto POST /my-index-000001/_forcemerge ``` The goal is to rewrite the segment files that have been created with Elasticsearch 6 (Lucene 7) to Lucene 8. Elasticsearch 8 can read Lucene 8 and Lucene 9 files. That's why this is needed. Unless you have an enterprise license which allows to read older indices directly and avoid all those manipulations. Hope this clarifies what is happening behind the scene 😉 ---
**Author:** ![Joao\_Barreto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joao_barreto/32/122272_2.png) [@Joao\_Barreto](https://discuss.elastic.co/u/Joao_Barreto)\ **Post date:** [March 4, 2024, 6:26pm UTC](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448/23 "2024-03-04T18:26:20Z")
What a day this was... The amount of tests I did to solve the problems with the Mapping... uffff... so many errors... Changes that I did into my Elastic 6.8 Mapping file: - Added "\_doc" section; - Added new "analysis" section with autocomplete\_filter and analyzer and a bunch of crap more...; - Added index.mapping.total\_fields.limit cause 1000 weren't enough. - Added include\_type\_name=true to my curl command when creating my test index And only after that, I could **manually** \_reindex my data from 6.8 into the new version. This was "basically" what you guys said. - Extract Mapping from Elastic 6.X - Create empty index with mapping of Elastic 6.X in Elastic 7.X - ReIndex previously restored index in Elastic 7.X, into the above index. - Create aliase so in theory we don't make developers cry. Bam... profit. Cost? I've become bald. 🤪 Next steps... I will need to script this entire process. The best thing in all this situation is that I only have to extract and "transform" 2 mappings, since I only have 2 main structures. 2 "main indexes". The 1000 indices that I have in the production environment are basically partitions. 1 per day. Anyway, I think for now, I'm all set. I wanna thank all of you who help me out during these past few days! Thanks for the help and have a nice week! ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 4, 2024, 7:59pm UTC](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448/24 "2024-03-04T19:59:11Z")
> [@Joao\_Barreto](#): > > - Create empty index with mapping of Elastic 6.X in Elastic 7.X > - ReIndex previously restored index in Elastic 7.X, into the above index. Why not reindexing directly in 8.x? ---
**Author:** ![Joao\_Barreto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joao_barreto/32/122272_2.png) [@Joao\_Barreto](https://discuss.elastic.co/u/Joao_Barreto)\ **Post date:** [March 4, 2024, 8:46pm UTC](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448/25 "2024-03-04T20:46:33Z")
Because I'm dumb as f.... 🤣 Since I've got the mapping situation right.... guess I could... try to reindex from 6 to 8... I'm gonna test that first thing in the morning. 😆 But I think for big indexes... like 4GB could give some problems... I dont know... but in theory it is way easier ---
**Author:** ![Joao\_Barreto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joao_barreto/32/122272_2.png) [@Joao\_Barreto](https://discuss.elastic.co/u/Joao_Barreto)\ **Post date:** [March 5, 2024, 10:23am UTC](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448/26 "2024-03-05T10:23:05Z")
Just to close this situation. I had to make a few changes in my mapping file, in order to use it in Elastic 8 - Removed "\_doc" - Removed "include\_type\_name=true" from the execution since it was removed in Elastic 8. And that was it... Now I think I'll need to build some Java program with Thread Pools, to speed up the process. Task pool: - List of Indices names. Task: - Step 1 - create index with mapping - Step 2 - Reindex - Step 3 - Refresh (?) 1000-1500 indices to migrate... this will be insane... ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [March 5, 2024, 10:38am UTC](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448/27 "2024-03-05T10:38:04Z")
Great! That's the best option IMHO when moving up to 2 major versions. You will immediately benefit from a lot of optimizations. If the data you are migrating is time based indices, I'd also add a force merge call to 1 segment at the end of your process (before the refresh). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 2, 2024, 10:38am UTC](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448/28 "2024-04-02T10:38:42Z")
Closed on April 2, 2024, 10:38am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/snapshot-restore-vs-reindexing-missing-documents-zero-failures/354448.md?page=1)