# How do Searchable Snapshot snapshots get cleaned up? **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-searchable-snapshots\ **Created:** [April 12, 2023, 12:08pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831 "2023-04-12T12:08:57Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [April 12, 2023, 12:08pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/1 "2023-04-12T12:08:57Z")
Hello All, I was curious if anyone knew the answer to the question: > How do searchable snapshot snapshots get cleaned up. To explain the question a bit more, I'll use the below example: I have: - A snapshot repository: `repo_A` - An SLM policy which take a global snapshot daily: `slm_A` - SLM policy points to `repo_A` - An ILM policy which uses content, hot, warm, cold (searchable snapshot), and delete: `ilm_A` - The cold phase points to `repo_A` - An index which is assigned to `ilm_A`: `index_A` As `index_A` progresses through `ilm_A` it will eventually make its way to the cold phase. At which point `ilm_A` will make a [clone](https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots.html#using-searchable-snapshots) (I think?) of the snapshot of `index_A` from `slm_A`'s global snapshot. This now leads to where my question is. At some point, `index_A` will reach the delete phase. Once here, `ilm_A` will now delete the local copy of `index_A` from the cluster. But what happens to the cloned snapshot for `index_A` that was made? Based on [this](https://www.elastic.co/guide/en/elasticsearch/reference/current/snapshots-take-snapshot.html#create-snapshots-different-time-intervals), `slm_A` won't clean it up because: > Since a policy’s retention rules only apply to its snapshots, a policy won’t delete a snapshot created by another policy. So, how do Searchable Snapshot snapshots get cleaned up? Does `ilm_A` do something to clean them up; will `slm_A` do something that isn't clear; or does the snapshot just never get cleaned up unless manually cleaned up? I wasn't able to find anything in the docs that went over this behavior, but if I missed something please let me know. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 12, 2023, 3:02pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/2 "2023-04-12T15:02:24Z")
ILM and SLM take separate snapshots, and each cleans its own ones up when they expire. The underlying data is shared through the regular snapshot-deduplication mechanism, but the snapshots themselves are logically separate and have independent lifespans. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [April 12, 2023, 3:17pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/3 "2023-04-12T15:17:18Z")
Thanks @DavidTurner for the info. So, in the above case, `ilm_A` actually takes another snapshot rather than cloning `slm_A`'s snapshot? And then when `index_A` hits the delete phase, `ilm_A` will both delete the local data, as well as the snapshot it took? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 12, 2023, 3:34pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/4 "2023-04-12T15:34:19Z")
> [@BenB196](#): > > So, in the above case, `ilm_A` actually takes another snapshot rather than cloning `slm_A`'s snapshot? Correct. Although even if it did work by [cloning a snapshot](https://www.elastic.co/guide/en/elasticsearch/reference/current/clone-snapshot-api.html) it would still be the case that the clone is logically separate with an independent lifespan. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [April 12, 2023, 3:45pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/5 "2023-04-12T15:45:59Z")
Thanks for the clarity on the topic. A follow up question based on this info, is there a way to have the lifespan of a searchable snapshot snapshot differ from the life span of an index on the cluster via ILM? If I wanted `ilm_A` to delete `index_A` from the local storage after 30 days, but wanted to keep the searchable snapshot snapshot for 90 days, is this possible? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 12, 2023, 3:58pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/6 "2023-04-12T15:58:55Z")
The `searchable_snapshot` ILM action takes a snapshot, mounts it as a searchable snapshot, then deletes the original data from local storage. This happens when moving to the `cold` or `frozen` phases. So it sounds like you want to move your data to `cold` or `frozen` after 30 days and then move it to the delete phase at 90. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [April 12, 2023, 4:05pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/7 "2023-04-12T16:05:44Z")
Somewhat, I want `index_A` to be moved to cold/mounted as a searchable snapshot. But I only want to keep `index_A` as a searchable snapshot (cold) for 30 days. After 30 days, I no longer want the index to be searchable on the cluster, but I want the snapshot that was used to still be around for another 90 days. The use case for something like this is a bit weird, but essentially in this case the cluster can only realistically leverage cold, and not frozen (because of some infrastructure limitations). So, I want to be able to leverage cold/searchable snapshot for a period of time, but still have a copy of the data to re-mount for an additional period of time if required/requested. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 12, 2023, 5:38pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/8 "2023-04-12T17:38:02Z")
> [@BenB196](#): > > cluster can only realistically leverage cold, and not frozen (because of some infrastructure limitations) I'm curious what those limitations are. IMO the simplest solution here is to use frozen (i.e. partially-mounted) searchable snapshots. Possibly instead of cold (i.e. fully-mounted) ones - it's pretty common to have a hot-and-frozen-only cluster. If you really want to keep the data out of the cluster, just use normal snapshots, maybe using SLM to manage them. If ever you want to mount some old data, clone your chosen snapshot by hand and then mount the clone. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [April 12, 2023, 7:26pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/9 "2023-04-12T19:26:31Z")
> I'm curious what those limitations are. There are 2 "limitations" on the infrastructure end. 1. The connection between the cluster and the backup repository isn't that great, only 300-400Mbps, for a cluster that generates a few TB of data a day. 2. The backup repository provider offers free egress which is why we use it, but they have a clause regarding egress. - You can't egress more data than you have stored in backups. While I don't think we'd hit this clause, I also don't want to get into a spot down the road where we do. Overall the main issue with using frozen is: > The cache is cleared when a node is restarted. If there was a way to have the above not happen, similar to cold, I think frozen would work. (Overall I do agree that if possible, frozen is a more ideal solution.) ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [April 24, 2023, 5:12pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/10 "2023-04-24T17:12:37Z")
A bit of an update here: I have found that I can leverage [`delete_searchable_snapshot`](https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-delete.html#ilm-delete-options) to keep a snapshot after its index has been deleted by ILM. The issue I'm running into now, is I'm unable to find a way to detect this now "orphaned" snapshot once this happens. It seems like the only place this information is ever stored is on the ILM side ([Prevent deletion of snapshots that are mounted and ILM delete action · Issue #73947 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/73947)), and looking over the [Get Snapshot API](https://www.elastic.co/guide/en/elasticsearch/reference/current/get-snapshot-api.html) it doesn't seem like any field tracks whether a snapshot is owned by ILM or SLM, this makes it a bit tricky to have something like a script come by and check for these "orphaned" snapshots. Anyone have any ideas/thoughts here? Note: I could use the snapshot name and parsing to find the date, but then I don't think I have a way of tracking whether it's managed by ILM/SLM, leading to potential issues mentioned in: [Prevent deletion of snapshots that are mounted and ILM delete action · Issue #73947 · elastic/elasticsearch · GitHub](https://github.com/elastic/elasticsearch/issues/73947) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 25, 2023, 7:30am UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/11 "2023-04-25T07:30:43Z")
> [@BenB196](#): > > The issue I'm running into now, is I'm unable to find a way to detect this now "orphaned" snapshot once this happens. Yes, I don't think this is a good approach. > [@BenB196](#): > > Anyone have any ideas/thoughts here? Yes, see my previous message: > [@DavidTurner](#): > > If you really want to keep the data out of the cluster, just use normal snapshots, maybe using SLM to manage them. If ever you want to mount some old data, clone your chosen snapshot by hand and then mount the clone. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [May 17, 2023, 12:15pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/12 "2023-05-17T12:15:28Z")
An update here, I've started to experiment with the Frozen/Partially Mounted snapshots at a small-ish scale, and this does currently seem to be a feasible solution. I did have one question before I started to expand this to scale\*. **Question:** Are there any more detailed documents on how the Partially mounted searchable snapshots work fundamentally? I was looking through the [docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/searchable-snapshots.html#searchable-snapshot-mount-storage-options), and there isn't too much about the technicals of how the feature interacts with the backend snapshot data stores (the docs seem to just cover a high level of how the local-cache works). I ask this question because the snapshot storage being used has the following limitations: - 1Gbps bandwidth shared between all nodes - 250 simultaneous TCP socket connections for each node In my small-ish scale testing I haven't noticed any issues with the above limitations, but I'm curious as I expand scope if I'll start to see issues. \* "At scale" is: 5k indices ~60TB of data that would be in the frozen tier. Majority of this data would be infrequently searched. (There would be 6 frozen nodes) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 17, 2023, 12:26pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/13 "2023-05-17T12:26:40Z")
Hard to say really, searchable snapshots are designed to use effectively-infinitely-scalable storage like S3. Whatever you're using isn't really compatible with S3 [in the sense that Elasticsearch requires](https://www.elastic.co/guide/en/elasticsearch/reference/current/repository-s3.html#repository-s3-compatible-services): > Note that some storage systems claim to be S3-compatible but do not faithfully emulate S3’s behaviour in full. The `repository-s3` type requires full compatibility with S3. In particular it must support the same set of API endpoints, return the same errors in case of failures, and offer consistency and **performance at least as good as S3** even when accessed concurrently by multiple nodes. It works on the same underlying files as regular searches, reading them in 16MiB blocks. So a single search that requires 250 random reads against a cold cache would involve 250 API calls, exceeding your request limit basically straight away. It'd also download ~4000MiB of data. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [May 17, 2023, 12:29pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/14 "2023-05-17T12:29:44Z")
Thanks for the info regarding how it works. I'll keep this in mind as I test a bit further with usage patterns that might run into this limitation. ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [May 17, 2023, 12:40pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/15 "2023-05-17T12:40:24Z")
Just adding a quick note: I had originally in my post: - 250 S3 API calls per minute for each node This was actually incorrect, the real limitation is: - 250 simultaneous TCP socket connections for each node ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 17, 2023, 3:54pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/16 "2023-05-17T15:54:36Z")
Oh ok, I believe the Amazon SDK imposes a limit of 50 simultaneous connections. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 14, 2023, 3:54pm UTC](https://discuss.elastic.co/t/how-do-searchable-snapshot-snapshots-get-cleaned-up/329831/17 "2023-06-14T15:54:43Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.