# Read segments directly from the snapshot **URL:** \ **Category:** Elasticsearch\ **Tags:** snapshot-and-restore\ **Created:** [June 2, 2023, 10:18pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122 "2023-06-02T22:18:26Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Ariel\_Zach](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Ariel\_Zach](https://discuss.elastic.co/u/Ariel_Zach)\ **Post date:** [June 2, 2023, 10:18pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/1 "2023-06-02T22:18:26Z")
Hello, I'm trying to read Lucene files (segments) directly from the files generated in the snapshot, but I'm encountering errors. Could you guide me on how to do it? Thank you. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 3, 2023, 2:52am UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/2 "2023-06-03T02:52:47Z")
Why do you want to do such a thing? It's definitely not supported and you should only use the Elasticsearch APIs. ---
**Author:** ![Ariel\_Zach](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Ariel\_Zach](https://discuss.elastic.co/u/Ariel_Zach)\ **Post date:** [June 3, 2023, 2:45pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/3 "2023-06-03T14:45:53Z")
Hello David, We currently have clusters with very large indices and high traffic. Reindexing is not a viable option as it performs poorly in these cases. It slows down over time, stresses the cluster, and is frequently canceled. Since each shard of the index is a file in the snapshot, Since each shard of the index is a file in the snapshot, the best approach is to read from there and insert the documents into a new index. I tried using the Lucene library, but I didn't succeed. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 3, 2023, 3:13pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/4 "2023-06-03T15:13:27Z")
> [@Ariel\_Zach](#): > > Since each shard of the index is a file in the snapshot The isn't the case. Each shard is made up of many files in the snapshot repository. > [@Ariel\_Zach](#): > > the best approach is to read from there and insert the documents into a new index. This is what reindex does, so I don't think it'll work any better. > [@Ariel\_Zach](#): > > I tried using the Lucene library, but I didn't succeed. I'm not surprised, the only realistic way to read data from a snapshot repository is by restoring it into Elasticsearch. ---
**Author:** ![Ariel\_Zach](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Ariel\_Zach](https://discuss.elastic.co/u/Ariel_Zach)\ **Post date:** [June 3, 2023, 4:04pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/5 "2023-06-03T16:04:44Z")
> The isn't the case. Each shard is made up of many files in the snapshot repository Where can I find this internal structure? Is it documented somewhere? > This is what reindex does, so I don't think it'll work any better. That's true, it does work, but in a 40TB index, it's useless. As I mentioned before, in a high-traffic cluster, the reindexing process gets canceled due to errors and is slow. > I'm not surprised, the only realistic way to read data from a snapshot repository is by restoring it into Elasticsearch. "Realistic way," if Elasticsearch can do it, anyone can do it. That's why I'm asking for help here. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 3, 2023, 4:46pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/6 "2023-06-03T16:46:43Z")
> [@Ariel\_Zach](#): > > As I mentioned before, in a high-traffic cluster, the reindexing process gets canceled due to errors and is slow. What is it you are looking to achieve by reindexing? What type of load is the cluster under? Is it only queries or a mix of queries, inserts and updates? How many shards does this index have? What is the average shard size? ---
**Author:** ![Ariel\_Zach](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Ariel\_Zach](https://discuss.elastic.co/u/Ariel_Zach)\ **Post date:** [June 3, 2023, 5:46pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/7 "2023-06-03T17:46:39Z")
Hi Christian, Cluster information: - 150k RPM reads - 200k RPM writes - It's a mix of queries, inserts, and updates. - 32 shards, each with 1.9TB, a total of 15 billion documents. > What is it you are looking to achieve by reindexing? The index is already too large, and we want to split it into multiple chunks. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [June 3, 2023, 6:48pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/8 "2023-06-03T18:48:00Z")
Reindexing is going to take a long time whichever approach you take, and given that you are having a lot of inserts and updates it might be be difficult to ensure consistency across the two indices. Have you looked into using the [split index API](https://www.elastic.co/guide/en/elasticsearch/reference/8.8/indices-split-index.html)? This should be a lot quicker and require less downtime. I do believe it does require a good amount of extra disk space though, but it would be worth considering. ---
**Author:** ![Ariel\_Zach](https://avatars.discourse-cdn.com/v4/letter/a/a88e57/32.png) [@Ariel\_Zach](https://discuss.elastic.co/u/Ariel_Zach)\ **Post date:** [June 3, 2023, 7:25pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/9 "2023-06-03T19:25:18Z")
That's something we took into account. The problem is that the index was created with version 6.8 and the split feature cannot be used. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [June 4, 2023, 7:42am UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/10 "2023-06-04T07:42:20Z")
> [@Ariel\_Zach](#): > > Where can I find this internal structure? Is it documented somewhere? Yes, although it's pretty complex and not at all focussed on this kind of usage - see [org.elasticsearch.repositories.blobstore package summary - elasticsearch 8.11.1 javadoc](https://javadoc.io/doc/org.elasticsearch/elasticsearch/latest/org.elasticsearch.server/org/elasticsearch/repositories/blobstore/package-summary.html) (NB varies by version). And all the classes to which those docs link. But then you'll likely need various other customisations to Lucene to actually open these indices outside of ES, and I don't think this is documented well. > [@Ariel\_Zach](#): > > in a high-traffic cluster, the reindexing process gets canceled due to errors and is slow. You're proposing doing the work in a separate process, isolated from this cluster, which seems like a reasonable workaround, but why not have this separate process be another Elasticsearch cluster dedicated to the reindexing work? > [@Ariel\_Zach](#): > > "Realistic way," if Elasticsearch can do it, anyone can do it. That's why I'm asking for help here. It's true that anyone can in theory duplicate all the mechanisms in ES needed to read a snapshot, but I think it's an unrealistic amount of effort to do so. ---
**Author:** ![Bob\_Penrod](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bob_penrod/32/116702_2.png) [@Bob\_Penrod](https://discuss.elastic.co/u/Bob_Penrod)\ **Post date:** [June 9, 2023, 10:46pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/11 "2023-06-09T22:46:00Z")
Back when our app used Lucene embedded in it, we used [GitHub - DmitryKey/luke: This is mavenised Luke: Lucene Toolbox Project](https://github.com/DmitryKey/luke) to inspect the Lucene files directly. It's evidently become an official Lucene module now. Reading TBs sounds like a lot of work. Good luck! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 7, 2023, 10:46pm UTC](https://discuss.elastic.co/t/read-segments-directly-from-the-snapshot/335122/12 "2023-07-07T22:46:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.